Working with Beanstalk Queues

Queueing is a natural part of modern web applications. Without it, you will end up with a poor user experience due to unnecessary execution blocking leading to inadequate response times. In Laravel (which everyone is using, of course), one of the easiest providers to get going with queues is beanstalkd. You can set it up in 15 seconds on any linux distro – and away you go without any configuration necessary. However when you need to debug and investigate, it isn’t really clear how to proceed. As such, I wanted to quickly go over some tools that I’ve discovered which are very helpful.

CLI based tube inspection

If all you have is a CLI, and you want a hassle-free way to check your tubes, I highly recommend beanstool. With no setup necessary, it will quickly give you all the info you need.

Web based tube inspector

If you need a GUI and/or more detail, then beanstalk console has you covered. It’s a very simple and lightweight PHP application which will give you plenty of info, and allow you to quickly and efficiently go through your tubes if they are clogged with several jobs.

Clear a queue in Laravel

This is perhaps the single most useful one. The scenario is that you have hundreds or thousands of faulty jobs stuck in your queue, and you want to quickly clear all of them. Amazingly, there is no trivial out-of-the-box way to do that, so I’ve come up with the following Laravel Command to get that job done.

The Laravangular Stack (with Postgres)

Let’s start March with some tech talk.

I’m going to be talking about technologies that we use to develop new web applications from scratch. For normal applications (ie. the 98% thereof which contrary to what hipster devs on your team believe, do not have a legitimate use case for mongo), this is the technology stack which I feel is far superior to any other choice. I call it the Laravangular Stack.

To begin with, all modern web applications must be designed API first with a separate backend and frontend. The reasons for this are obvious;

  • Separation of concerns
  • Easier automated testing
  • Code once for multiple consumers (ie. web, mobile, device, API for 3rd parties, etc)
  • Performance
  • Easier to upgrade components
  • Easier to debug
  • Easier to divide dev workload
  • Your application architecture isn’t a sack of shit
  • Ability to have a rich, interactive frontend (and stop pretending that jQuery can accomplish this)

The question then becomes, what specific technologies do we use for such an application? In my view these answers are obvious.

Laravel has you covered on the backend
Laravel is the best thing that’s ever happened to web programming period. The most popular framework has you totally covered for API-first projects, and will deliver more features with more simplicity than anything else in existence – while the community is hard at work churning out thousands of packages for you to leverage in your project. It’s an obvious choice.

Angular for a superb user experience
Angular, at least for now, is the only proper actual front-end framework. Yes, it’s far from perfect, and some have even gone so far as to call it a tech demo…. however until Angular 2 comes out, there isn’t anything which can compete with it because everything else is essentially a bit of wet lettuce. Frontend JS frameworks are a very recent invention, and it’s an interesting new space – but the sooner you come to terms with the fact that Angular at the moment simply shits on everything else out there, the sooner you can begin to create world-beating applications.

PostgreSQL for RDMS
Many people still use MariaDB (or god forbid MySQL), however I’ve felt for a long time that it is rather inferior to PostgreSQL. A project I’m working on now has cemented my view, with 2 rather sharp examples.

Our first problem stemmed from having a very high write workload for one logging aspect in our system. The use case involves a very large number of external devices writing logs to our system every few minutes. In our testing, InnoDB is a severe bottleneck for write-heavy workloads. I have read up on this, and the tl;dr is that there is no way to fix it – it’s just the way it’s designed. That rather sucks.
The second problem are UUIDs. UUIDs are pretty much the gold standard now, especially when it comes to anything API. How do you store them in a database though? Well, in the case of MariaDB, there is no perfect solution. If you want optimal performance, you can choose to store it as a binary 16 – and say goodbye to ever being able to manually read or write any UUID. Alternatively you could store it as varchar 36 (or 32 without the dashes), but take a non-trivial performance hit.

PostgreSQL on the other hand has had a naitive UUID field for a while now, which is internally stored as if it were binary, but behaves like a char. Likewise, it has no performance problems with write-heavy workloads. Perfect!

Of course when you come down to the bottom of it, there are many more differences between MariaDB and PostgreSQL – and you’ll likely find that PostgreSQL can do everything MariaDB does, but also more. It just so happens that PostgreSQL is always first to the party with any number of features that it’s had first in the last decade or more, including being an actual database system with full ACID compliance. It’s only pitfall if any is the lack of tools (relatively speaking) compared to MariaDB. If you can get past that barrier however, I would definitely recommend it.

Everything else is irrelevant
Whether you use Apache or Nginx, they both proxy pass PHP requests straight to php-fpm in all modern configurations, and let’s face it, nobody serves static assets anymore – it all comes from CDNs. Whatever other technologies used, wherever you are hosted – it’s less relevant than getting the fundamentals right.

Laravel Groupwise Max

For a project I’m working on, I really needed to do a groupwise max in Laravel. If you’re here, you likely don’t need an explanation of what that is, so much as how to actually bloody do it, so without further delay;

Let’s say that we have Actions, and we have an ActionLog, and you want to get the latest ActionLog entry for a series of Actions – determined by their “timestamp” field. You would write this:

Unfortunately there’s no way to do it without the dreaded DB::raw, but otherwise it turns out to be pretty simple.

Phunconf 5.0

Just attended Phunconf 5.0 yesterday. It was a great event with lots of very interesting discussions, however probably the most salient point I want to draw attention to is the following image.

To provide some context, at Phunconf events our tradition is to open the floor to the community to suggest topics for circle discussions. This year, all attendees were given two red circles, which they could put on any topic of their choice, in order for everyone to democratically pick the most popular topics which we would then discuss. The outcome was as follows:

Phunconf 5 Whiteboard

The two post-it notes with so many circles on them that you can’t see what’s written there, are of course Angular and Laravel.