Blog

The return of the database

Posted by Michael Cullum in Website on August 9th, 2015

Any of you who have tried to access phpBB over the past few days might have experienced a variety of different errors, most notably one about being unable to find the database `phpbb`. This is the full, slightly more technical post mortem of the issues highlighted in the announcement here.

Infrastructure

Firstly, some background on our infrastructure. It is hosted very generously and free of charge by Oregon State University Open Source Lab [OSUOSL or OSL] which is funded by donations and grants from individuals and large organisations (such as Google). We run our systems across a number of virtual machines (VMs), some of which are hosted on our own dedicated machines. Others were moved onto the main OSL cluster back in January as our old dedicated machines became more unreliable and unstable. We also run a number of services on centralised and function dedicated machines by the OSL such as their database servers, mailing list instance etc. which are used by a number of projects. They also very generously manage many of our VMs (using Chef), something which we’d like to make a blog post about in the future.

We now host all our databases on OSL’s main database cluster of two virtual machines served by a virtual ip which will use server SQL1 when possible or failover to SQL2 if SQL1 is down (which has happened a few times). SQL1 is their master (read & write) and SQL2 is their slave (Read-only) and SQL1 replicated to SQL2.

Replication Errors

On July 15th a number of issues with replication from SQL1 to SQL2  were noticed with some session tables which caused replication to be paused and a large number of statements were skipped. OSL then restarted replication but on the 16th we began to experience even more issues in far greater numbers than on the previous day and on the 20th a decision was made to entirely reload SQL2. Anticipating the only effect would be a bit of a slowdown on SQL1 due to a large number of reads, they went to do the maintenance at 00:00 on the 31st of July. The standard procedure for that is break replication, delete the databases off SQL2 one-by-one, and resync from SQL1.

 SQL1 Issues

At 00:20 SQL1 started experiencing errors and investigations began immediately. Despite the fact that it was believed to be a master-slave configuration between the two servers, it was in fact a master-master replication meaning although SQL1->SQL2 replication had halted, SQL2->SQL1 had not and SQL2 replicated a lack of data back to SQL1 causing the dropping of databases to occur on both machines. Immediately database dropping was halted and therefore some other project databases were unaffected but phpBB databases had already been dropped. As we had no valid failover to a read-only server (Normally this would be SQL2), OSL were then left to restore backups. However, the backups server had quite slow I/O and there were a lot of databases to restore (Just one of our databases is ~33GB and we have a number of databases and there are a number of other projects on the cluster). The backup restoration finished at 10:49 on the 1st August and took such a long time due to the aforementioned reasons and issues with problematic database structures. Once the backup restoration was complete the binlogs (essentially a log of all sql queries executed) were replayed to catch the backups up to just before the maintenance. The binlog replaying finished at around 04:22 and production databases then began to be moved back into production. More details on the SQL1 and replication issues can be read about on OSL’s own postmortem.

Maintenance Page on .com

Unfortunately, throughout most of this time we were just displaying an error saying that the `phpbb` database did not exist. Due to the time of year most of our team who would normally work on putting up (and then taking down once things were fixed) a maintenance page were away on holiday or leave (without internet or without their login credentials that they’d need such as ssh keys or sudo logins or on restricted internet connections not allowing SSH) and team members who did have access to the internet didn’t have the necessary access to repositories or our servers. Therefore, only information available about the downtime was from our twitter account and facebook page. For this we do sincerely apologise as we are aware many of you were unaware of why our site was down.

Missing Data

On Wednesday we began to realise there was some data missing after we discovered some posts had disappeared. Replication from SQL1 to SQL2 broke on the 15th July and we take backups from SQL2. The backup OSL had restored therefore was the one from the 15th July. OSL found the binlogs (/var/lib/mysql) on SQL2 and then replayed those in order to bring the state of the database back to the present. Unfortunately the command used to replay the binlogs only replayed the sql commands from the 17th July and this gap was only realised after OSL had blown away /var/lib/mysql on SQL2 in order to restart replication from SQL1 to SQL2 once we were back up and in production.

This means all actions done between the 15th and 17th July on any *.phpbb.com sites will have been lost.

Notifications

Also due to some unusual behaviour, everyone’s notifications settings on phpBB.com have been reset to never send out emails for notifications. You can still change your settings here but in order to ensure people don’t miss notifications, we’ve set everyone’s settings to now email for the following but you can of course revert this change if you so wish:

  • Someone replies to a topic to which you are subscribed
  • Someone quotes you in a post
  • Someone creates a topic in a forum to which you are subscribed
  • Someone sends you a private message

Into the future

OSL are implementing a number of policy changes and changes to their backup procedures (Read more here) in order to prevent this sort of thing from happening again, as are we. We understand many of you depend on phpbb.com for support as well as the resources it provides (downloads, documentation etc.). We will be looking at provisions to ensure that our infrastructure is less interdependent and removing single points of failure. This has also highlighted some rare edge-case bugs in phpBB which we are looking to patch as a matter of priority. We’ll also look at how we can better communicate downtime in the future, make appropriate maintenance pages easier to display, and ensure we always have people around who can deal with such situations.

Often when these kinds of situations arise, we receive questions about how our community can help us. While we appreciate the gesture of making a donation to us, the phpBB project does not accept financial donations. If you would still like to make a financial gift, you can support us indirectly by donating to OSUOSL. You can also help us directly by being active in our support forums, IRC, or by submitting source code patches. We are only here because of the community behind us, so anything you do to help the rest of the community helps us.

We do apologise once again for any problems this might have caused you.

As a note, all times are UTC+1 (British Time).

Symfony Live London 2015 – Please vote for phpBB!

Posted by Michael Cullum in Events on August 3rd, 2015

Symfony Live London and the rest of the SymfonyLive/SymfonyCon conference series are amazing conferences that bring Symfony developers together from around the world into one place for a day (or multiple days) of talks on a huge variety of Symfony related topics. Over the past few years, since our integration of Symfony into phpBB core and phpbb.com, our relationship with the Symfony community has grown. Many developers from their community now contribute to phpBB itself, something we have reciprocated in addition to sending many of our team members to Symfony conferences and meetups.

This year Symfony Live London will have a number of community sponsors of whom they will provide a sponsor booth (and much more) in order to directly communicate with many of the Symfony Community. Those projects are decided upon after the results of a public vote.

These sorts of events allow us to spread the word about phpBB and help get more people interested in working on the project, allowing us to develop phpBB to a better quality and do so faster. We are asking you, our community, to reach out and support us by voting for phpBB in their poll.

You can vote by clicking here.

Voting finishes in just a couple of days so please don’t wait. Should you wish to help support us further by spreading the word and garner us even more support, please do retweet us and share our facebook post.

Thanks!

Edit: Thanks to all who voted, we are proud to announce we have been accepted as a community sponsor of Symfony Live London 2015! If you’re in attendance do come over and say hello.

phpBB Developer Documentation

Posted by nickvergessen in Development, Extensions with the tags , , , on July 5th, 2015

When the phpBB Team met in Prague last month for a short productivity meeting we achieved many things. We fixed bugs, worked on our bamboo infrastructure, reduced the size of the extension validation queue and worked on a new phpBB/Symfony Authentication bridge however one of the most significant things we did was start the base of our new developer documentation:
https://area51.phpbb.com/docs/dev/31x/

The plan with this documentation is to consolidate all core and extension development related information that is currently spread out on our wiki, Area51 (and a variety of other places) into one central location.

As a first step the “[HowTo] Convert a 3.0 modification into a 3.1 extension” tutorial was converted and is now available to view here.

Since then, we have ported over many many other Wiki pages and readmes to the new doc system including:

The documentation is written using rst and our documentation source can be found in the “development” folder of our pre-existing documentation repository: https://github.com/phpbb/documentation/

We will be continuing to move existing documentation over the coming months and if you are familiar with rst and git(hub) and want to help, please do feel free to send pull requests. Whether it is with porting existing documentation over, fixing typos or even writing brand new pieces of documentation, every contribution is appreciated.

phpBB at Symfony Developers UK (London)

Posted by Michael Cullum in Events on May 25th, 2015

At a recent meeting of the Symfony Developers UK user group in London I gave a talk titled “How we Build phpBB: The Next Generation”.

Screenshot 2015-02-03 01.33.28

After a brief introduction to phpBB’s functionality the talk covers different aspects of phpBB’s development. It provides insight into phpBB’s history, in particular that of its versioning, and how this affects how we operate to this day – keep in mind that we published our new release policy after this talk was presented.

The talk follows the process a feature goes through from a phpBB user’s idea all the way to making it into a phpBB release. Along the way it covers the numerous different tools used to aid us in developing phpBB; such as the issue tracker, GitHub, Bamboo, Travis CI. These tools are vital in helping our quality assurance process preventing regressions and they make it easier to develop phpBB. The talk concludes with a summary of the Symfony components we have adopted in phpBB and how we use them. It further explains why we are using more external libraries and components instead of “reinventing the wheel” as the term goes.

The talk was well received by the audience and many members of the Symfony community were positively surprised by the amount of Symfony components we now use in phpBB, by the free code and security audits performed by the styles and extensions teams on submitted customisations in order to ensure a high level of quality, and the outstanding security record that phpBB 3.x has now gained.

A video of the full talk can be found on Youtube and the slides are available on Speaker Deck. Please note that the slides may not make sense without the context of the talk.

If you run a user group or conference and are interested in having this talk, or a similar one, presented at your event by myself or another phpBB team member, please contact me or a member of the management team.

[Poll] Community Choice: Next Official Extension

Posted by VSE in Extensions with the tags on May 13th, 2015

The Extensions Development Team would like to ask you (again) to vote for another official extension!

Our previous community choice resulted in our Auto Groups and Pages extensions. We have a few more ideas and would like the community to help us decide which of them would be the most popular:

  • Advertisement Management – Ability to add multiple advertisements and display in various locations (header, footer, specific forums, etc.)
  • Google’s new reCAPTCHA – spam countermeasure, more information
  • Paypal Donations – Add donations with option to add users to groups based on payment
  • Welcome PM – send new users a customised welcome private message

Vote Now to be heard (until May 19)!

OSCON 2014 Recap

Posted by darcie in Events with the tags on August 9th, 2014

Last week phpBB again attended OSCON, the O’Reilly Open Source Convention, in Portland, Oregon. Yuriy, Derk, and I had a great time chatting with those attending the conference as well as those exhibiting. We were provided booth space by Bluehost as part of their ongoing commitment to support open source projects. We had a great spot right at the front of the hall.

2014-07-23 10.07.24_1 2014-07-22 12.35.47

We arrived and set up Sunday in preparation for the Expo Hall opening Monday. Monday evening, the Expo Hall hosted a one hour first look at all the booths for conference attendees. The hour went by quickly, but after parties and further networking opportunities closed out the day. The “Elements” themed party the first night had us flying high and ready to conquer the week!

2014-07-21 17.36.02 2014-07-21 17.35.03

Tuesday started damp, as we would expect for Portland. This was the first full day with the exhibitors, with that night being booth crawl night when the convention center brought in food and drinks. Bluehost had Tim O’Reilly come play a game of ping-pong with them!

2014-07-22 18.40.26 2014-07-22 18.40.04

All week, we explored the hall and networked with other groups and attendees. We found our name on the Google Summer of Code board and chatted with the representatives of the program. We talked with someone in Community Development at O’Reilly Media about partnering with them on some interesting opportunities. And we left our mark on the giant chalkboard in the convention center hallway.

2014-07-22 11.20.55 2014-07-22 17.40.37_1

Bertie helped give away lots of gummy bears. We talked with many people about 3.1 and improvements due in the next version. We had a great response to the concept of extensions and responsive layouts. We met many conference attendees that mentioned they got their start in coding by having a phpBB forum as a teenager, which was very rewarding!

2014-07-22 21.02.09

Additional highlights of the conference are available at the OSCON 2014 site.

[Poll] Community Choice: Next Official Extension

Posted by EXreaction in Extensions with the tags on July 15th, 2014

The Extensions Development Team would like to ask you to vote for the next official extension!

We’ve compiled the following list for you to choose from:

  • Auto Groups – an extension which allows admins to configure conditions, e.g. reaching defined post count, membership days, or warnings, that will add users automatically to a defined usergroup.
  • Posting Templates – an extension that allows admins to create predefined/canned message templates that can be used when a user starts a new topic.
  • Paypal/Donation – a page for accepting donations through PayPal including the ability to add users who donate to a special group (Donor Group).
  • Pages – an extension that allows admins to create custom static pages for their forum.

Vote Now to be heard (until July 28)!

Development Update: Notifications & Custom Profile Fields

Posted by naderman in Development with the tags on February 19th, 2014

After releasing Alpha3 last week, the development team is working on fixing outstanding bugs and getting ready for the first Beta release. As I mentioned in a previous blog post, we are now working on transitioning existing hard coded profile fields – such as occupation or location – to custom profile fields. The pull request converting location to a custom profile field also introduced a new option to display custom profile fields on the memberlist. As we are turning hard coded fields into custom profile fields, we are ensuring that all previous functionality can be achieved through configuration. At the same time these options can now be used for other custom profile fields you may wish to display.

Ascraeus Notifications Dropdown

Ascraeus Notifications Dropdown

As we fixed a bug regarding the display of avatars in notifications I want to use this opportunity to show you the new notifications functionality we built into phpBB Ascraeus 3.1. You may be familiar with the concept of a central location for notifications from Facebook or Google+. The idea is to provide users with a single location that will inform them about all the activity they are interested in. So far this includes notifications regarding posts, such as replies to bookmarked or subscribed topics or posts you have been quoted in, notifications regarding moderation of newly submitted posts waiting for approval and reported posts as well as notifications for new private messages and requests to join a group you lead. You can configure whether and how you would like to be notified for each of these notification types individually in the User Control Panel. The options are email, Jabber or the notification on the website. Extensions can easily add new notification types making the system very flexible.

Further we’re now testing phpBB automatically with the alpha releases of PHP 5.6 on Travis CI. A final version of PHP 5.6 is expected this summer.

phpBB 3.1 Alpha3 Preview Release

Posted by naderman in Development with the tags on February 10th, 2014

We’re happy to announce the immediate availability of the third preview release of the upcoming phpBB 3.1, codenamed Ascraeus.

This preview release is targeted at phpBB developers and advanced users who wish to assist us with this very important testing phase. We do not recommend, and will not support, running this version in a live production environment.

As with previous releases we compiled a list of feature and bugfix highlights on our wiki. We now have a generic framework for dropdowns in prosilver. We’ve started to use it for the new topic tools button from where you can manage bookmarking, attachment downloads and subscriptions to topics. Topic titles now link directly to the oldest unread post.

The only known issue remains that updating from 3.0 to 3.1 on Microsoft SQL Server is not possible.

Please use this release for testing purposes only. As always, we will provide no support for upgrading until the RC stage and urge users not to use testing releases in a production site. Please help us by reporting any bugs you find via a ticket on our tracker.

Unstable packages are available on Area51 at area51.phpbb.com/downloads/. Information on moving from 3.0 to 3.1 (currently only for testing purposes) is available at wiki.phpbb.com/PhpBB3.1#Moving_from_3.0_to_3.1

Please note that phpBB 3.1 has some requirements that differ from phpBB 3.0:

PHP version 5.3.3 or greater
PHP’s json extension
Only PostgreSQL versions 8.3 and above are supported

Stuttgart Developer Meetup

Posted by naderman in Development, Events with the tags on February 5th, 2014

After the German phpBB Association’s general meeting on Saturday a group of developers spent Sunday making some great progress toward a first phpBB 3.1 Beta release. The German phpBB Association runs phpBB.de – one of our many international support sites. A number of German phpBB developers – including me – are members of the German association and decided to use the opportunity to work face-to-face for a day. We were joined by phpBB.de volunteers who helped with bug triaging and pull requests and worked on translation related projects.

Sunday group picture of phpbb.com and phpbb.de team members

Sunday group picture of phpbb.com and phpbb.de team members

We managed to merge the password hashing manager which will allow us to easily transition to newer more secure password hashing algorithms in the future. It will be the basis for an easier conversion process from hashing algorithms used by other forum software. Currently switching from another forum software to phpBB requires the use of a custom authentication plugin to keep passwords working.

The refactoring of profile fields which was merged at the event will allow extensions to work with profile fields and add new types of profile fields. This is a first step toward modularizing the entire profile so that all default fields can be configured by administrators.

The group working on phpBB in Stuttgart

The group working on phpBB in Stuttgart

After a lot of debugging over the past weeks we finally worked out why our tests were failing on PHP 5.5. It turned out that PHP 5.5.7 which is used on Travis CI has a bug in the OPcache extension. The OPcache extension is the new default opcode caching mechanism in PHP 5.5 which greatly speeds up the execution of PHP code.

On our path to Alpha3 – hopefully the last Alpha release of phpBB 3.1 – we also spent some time working on bugs in the build process. These prevented us from building packages for Alpha3 so far. As a consequence we will stop providing old text format MOD files for style and language file changes. We realized the algorithm may generate incorrect results which could go unnoticed, resulting in incorrectly updated styles or language files. Instead we will provide regular patch files generated with GNU diff. Alpha3 will contain all the improvements we finished over the weekend.