StubHub All Access » News

Recap of StubHub’s First Annual Hackathon

Rashid Aidun — Tue, 18 Jun 2013 18:54:28 +0000

StubHub’s first ever hackathon was held on the weekend of May 17th. We had over 25 participants join us for the opening social hour Friday evening where Director of Software Development, Mehdi Ghazizadeh, gave an introductory speech before two full nights of hacking.

Participants socialized briefly then formed teams. Most teams worked through both of the nights. By Sunday morning, four teams of various sizes presented ideas for prizes. Though the rules did not require it, all of the teams ended up using the StubHub and Zvents APIs.

The final presentations showcased awesome experiences around events:

First place ($5,000): Scheme Zone – an application to organize “schemes” such as “the wave” at events. Collaboration involved text chat, a group video chat, and a seat map with all participants.

Second prize ($3,000): Take Me Out – A dating application around events. Singles would log in, choose events, and the application would pair them up for a date to the event.

Third prize ($2,000): MerchPerch – A team of six formed this merchandising experience for both sellers and buyers. Sellers had access to inventory management tools. Buyers had access to a web application to purchase merchandise at events without needing to stand in line. This team stood out on formal engineering process — they created requirements, then wireframes, then formed 3 teams of 2 to code the various components. They asked some of our engineers for API and DataModel help.

No prize: Rota – A music discovery application. Unfortunately for this team of two, by presentation time, very little outside of a nonfunctional menu was complete.

It was a very energizing experience for everyone. Many thanks to the StubHub Boston engineering along with many of the folks from the San Francisco office for help making this hackathon a reality.

Stay tuned. More StubHub hackathons to come! Visit us on Linked In to learn more about job opportunities at StubHub.

Behind ths Scenes with Eurosport

jackie tincher — Tue, 18 Jun 2013 03:40:15 +0000

A great Behind the Scenes StubHub UK All Access feature with the crew from Eurosport the Aegon Championships.

Click here to view more.

In April, StubHub announced a new UK sporting partnership with the Lawn Tennis Association (LTA), making StubHub the official ticket marketplace for three major UK grass court tennis events: The Aegon Championships at The Queens Club, The Aegon Classic, Birmingham and The Aegon International, Eastbourne.

Last week saw the first of these partnerships, The Aegon Championships at The Queens Club taking place. The tournament is widely regarded as the men’s warm up tournament to Wimbledon and we’ve been behind the scenes to share daily access at the Championships.

Behind the Scenes: Meet the Aegon Ball Girls

jackie tincher — Tue, 18 Jun 2013 03:32:50 +0000

StubHub All Access: Ball girls. So what’s it like being a ball girl at the Aegon Championships? With StubHub UK All Access we were able to go Behind the Scenes and find out.

Check out this video with the girls!

Behind the Scenes with the Aegon Transport Crew

jackie tincher — Tue, 18 Jun 2013 03:26:42 +0000

The StubHub UK All Access feature takes us “Behind the Scenes” with the transport crew at the Aegon Championships.

Click here to view this exclusive video!

Behind The Scenes: Head Groundsman

jackie tincher — Fri, 14 Jun 2013 23:41:25 +0000

Today is day four of The Aegon Championships at The Queens Club and we’re pleased to share with you another exclusive glimpse behind the scenes at all the action!

Do you ever wonder how much work goes into the Aegon Championships courts which are widely regarded as the best in the world? Our StubHub UK All Access feature goes Behind the Scenes to speak to The Queen’s Club groundsman Graham Kimpton.

Click here to view this exclusive video of what it takes to make these courts world class.

For tickets, please check StubHub.co.uk

Behind the Scenes with David Law, Aegon Media Director

jackie tincher — Thu, 13 Jun 2013 00:37:02 +0000

Today is day three of The Aegon Championships at The Queens Club and we’re pleased to share with you another exclusive glimpse behind the scenes at all the action!

Today, Di Dougherty catches up with the Eurosport crew, who broadcast daily throughout the tournament. Watch our exclusive behind the scenes chat with Media Director, David Law.

This week sees the first of these partnerships, The Aegon Championships at The Queens Club taking place. The tournament is widely regarded as the men’s warm up tournament to Wimbledon and we’ve been behind the scenes to share daily access at the Championships.

Check here on StubHub.UK for Aegon Championship Tickets.

Behind the Scenes: Stringers at the Queens Club

jackie tincher — Tue, 11 Jun 2013 21:38:20 +0000

Today is day two of The Aegon Championships at The Queens Club and we’re pleased to share with you another exclusive glimpse behind the scenes at all the action!

Today, Di Dougherty interviews the Babolat stringing team, who work day and night to ensure that the players’ rackets are tuned to perfection.

Click here to view this exclusive glimpse at the Stringers at work.

For tickets, please visit StubHub now!

The Toyota 5 Whys

Quan Ding — Tue, 11 Jun 2013 00:40:04 +0000

I was reading the book “The Lean Startup” the other day and learned about this problem solving technique called “the Toyota 5 whys“. You can google it for the detailed description. Basically when you are trying to find the root cause of a problem, don’t settle when you’ve identified the cause. Instead, repeat the process to see if you can find the cause of the cause. The theory is, if you iterate this process five times, you’ll find the true root cause that you should really fix.

Here’s an example taken from the Wikipedia:

The vehicle will not start. (the problem).

1. Why? – The battery is dead. (first why)
2. Why? – The alternator belt is not functioning. (second why)
3. Why? – The alternator belt has broken. (third why)
4. Why? – The alternator belt was well beyond its useful service life and not replaced (fourth why)
5. Why? – The vehicle was not maintained according to the recommended service schedule. (fifth why, a root cause)
6. Why? – Replacement parts are not available because of the extreme age of the vehicle (sixth why, optional footnote)

Therefore, instead of just replacing the battery, what should be done is maintenance of the vehicle according to the maintenance schedule. If you just replace the battery, it will die pretty soon again since the alternator belt is still broken.

I realized that this technique is very useful in fixing software defects as well and that I was actually doing it subconsciously. Here are a couple of real life examples.

The first is when some time ago QA assigned me a bug about a NullPointerException and asked me to add null check before dereferencing the variable. Sounds easy, right? Not so fast. Looking closer at the code where the NullPointerException was thrown, I saw two problems that a null check won’t fix. First issue was that the code block encrypts some sensitive data. By just adding a null check, the code will skip the encryption, which is probably not what we want. Second issue was that the value is one of the predefined Enum constants, which shouldn’t be null to begin with. Tracing back the call stack, it turned out that there’s a new constant added to the Enum class. But the code that translates a string value into a Enum constant hasn’t been updated to handled the new enum value. So the translation code returns a null if the string maps to the new Enum value. That’s the root cause of the NPE we should fix. If we just add a null check around the code where NPE is thrown, more than likely we will just end up with a NPE again because the null value is still there and will probably be used somewhere else. Using the 5 whys approach, the process will look something like this:

The program throws a NullPointerException (the problem)

1. Why? – Because the code dereferences an Enum variable whose value is null.
2. Why? – (is the value null)? Because the code that translates a string into an Enum returns null for the given input.
3. Why? – Because a new constant was added to the Enum class but the translation code hasn’t been updated.

There’s no “5 whys” but you get the idea.

In another case, I was working on a production bug. The issue was that the operations team noticed a sustained 100% CPU usage in the API servers. The operations team has done the initial triage using the java thread dump. They found that the issue was caused by multiple threads accessing a HashMap, which is a static member of a class, for both reads and wites. HashMap is not thread safe. So the recommendation was to either add synchronization or use ConcurrentHashMap instead. But locking/synchronization is the evil of all scalability issues. So I’m not quite satisfied with the solution. First I wasn’t convinced that concurrency issue can lead to CPU spin. More often it’s linked to thread liveness issues (dead locks for example). But the operations people showed me some links which point out that the implementation of the HashMap DOES have a busy while loop in it (not in Openjdk 1.6 though as I checked the source code). Then I wondered what the HashMap was used for. After reading the code, it appears that the HashMap is used in a JMS listener to track the number of times a particular message has been delivered. If the count goes over a predefined threshold, the message will be ignored. So it’s essentially a in-house implementation of duplicate message detection mechanism. Unfortunately this implementation is broken. It only works in the case of a single JVM because it uses a static HashMap to store the count. But our deployment has multiple application servers and the code won’t be able to count the messages that are sent to other JVMs. Then my follow up question was why do we roll our own implementation after all? Doesn’t ActiveMQ, our message queue system, already have the duplicate detection feature built-in? So it turned out the real fix was to remove the HashMap code completely and let ActiveMQ handle the duplicates detection. Again, applying the 5 whys technique, the process will look something like this:

API server has 100% CPU usage (problem)

1. Why? Because the JMS listener code uses HashMap which isn’t thread safe.
2. Why (use HashMap)? To track the number of times a message is delivered and detect duplicates
3. Why? Because the developer wasn’t aware of the fact that ActiveMQ has this support built-in

As you can see in the examples, jumping to a solution too quickly will lead to a premature fix which does nothing but hides the real problem. The real problem will probably just resurface in a different form until it’s fixed. So next time when you think you’ve found the fix, stop and think for a moment. Ask yourself (or whoever appropriate) a few more whys to see if there’s some deeper issue lurking around.

Not everyone believes in this method though. Check out the criticism section of the 5 Whys on Wikipedia, which I mostly agree. But that’s not to say there’s no value in this method. People have the tendency to stop whenever they think they’ve solved the problem. Having this 5 whys mentality helps you overcome this tendency and get closer to the true root cause of a problem.

Last but not least, I have to admit that I haven’t applied the 5 Whys methodology to the true purpose of this method, which is to identify the process failure, or the lack of a process. “people do not fail, processes do”. So in the first example, we can keep asking whys and the real fix maybe that our QA or unit test process should be improved to catch the problem. In the second example, maybe our code review process is lacking. These are bigger issues that require a lot more collaboration and team effort. It’s a work-in-progress as we are marching toward the continuous delivery model.

P.S. StubHub is HIRING! Come and join us!

Writing Code that is Unit Test Friendly

Quan Ding — Mon, 10 Jun 2013 23:13:02 +0000

Unit test code is probably just as important as the production code. Not only does it prove that your production code works as expected, it also serves as the regression suite to ensure the feature is not broken in the future. I would argue that there should be more test code than production code. With that in mind, it pays to write your code in a way that’s easy to write unit tests for.

However, if you are not following the TDD approach and only start writing unit tests after you have finished coding for the feature you’re implementing, you may end up having a hard time to test your code because of the way it’s written. Here are some tips to avoid the heart attack when you start writing unit tests and find that impossible. The tips are all derived from problems I’ve seen in our real production code.

1. Make sure the result of the method invocation is visible either through the return value or the input arguments.

Here’s an example taken out of our production code that goes against the rule above:

public void doSomething(ValueObjectClass vo) {
// fetch the domainObject using vo and do something
... domainObjectDAO.save(domainObject); }

The method doesn’t return any value. All input arguments are acting as value objects, meaning they are used only to transfer values. That means from outside of the method, nothing has changed. The change is materialized in the domainObejctDAO.save() call. In unit test code, domainObjectDAO is often a mocked object (which is the right thing to do since we want to decouple our code from external dependencies.). This makes is very difficult to verify that doSomething() is doing the right thing because we have nothing to verify against. We can’t check return values because there’s none. We can’t check the input argument value changes because they don’t change. And we can’t query database for value changes even if I want to because the mock DAO doesn’t save it into the database at all.

A better approach is to make doSomething() itself returns the modified object (or change the value of the input arguments, whichever is appropriate). Have another methodcallDoSomething() that calls doSomething() and domainObjectDAO.save(). So the unit test for doSomething() can verify the values are correct. And the unit test forcallDoSomething() can verify that both doSomething() anddomainObjectDAO.save() are invoked, assuming both doSomething() anddomainObjectDAO.save() are tested separately by other unit tests.

An even better approach, if you follow the domain driven design principle (DDD), is to move doSomething into your domain model (instead of the domain models being anemic with nothing but getters and setters). So you test your domain model’s doSomething() method without worrying about how it’s persisted. And write a service layer code which will call your domainObject.doSomething() as well as DAO.save(). To test the service layer code, you only need to verify that both doSomething() and save() are invoked. You can read more about anemic domain model here. Unfortunately, almost all “domain” objects I see in our production code are anemic.

If you are unlucky that you have to deal with testing behavior that’s not visible externally as in the example I gave above, fortunately Mockito has a feature called Captor that allows you to capture the arguments that are passed to a mocked object. You can use Captor to verify the values of the domainObject. Here’s how you test the example above:

//Alternatively, there’s also a @Captor annotation that works at class level (not within the method) ArgumentCaptor argument = ArgumentCaptor.forClass(DomainObjectClass.class);
verify(mockDAO.save(argument.capture()) DomainObjectClass domainObject = argument.getValue();
//Do some verification of values in domainObject Assert.assertEquals(domainObject.getSomething(), “It works!”);

2. Don’t use static method if you can avoid it

This is not a hard rule, more of a limitation of the Mockito testing framework as it doesn’t support mocking of static methods. Checkout their FAQ for Mockito’s limitations. There are some test frameworks that support mocking of static methods. But based on my personal experience, Mockito is the most straightforward to use so far. So I would stick to it for as much as you can.

So why should I not use static method and what do I do if I’m testing legacy code that has static methods? Here’s in Mockito’s own words:

Can I mock static methods?
No. Mockito prefers object orientation and dependency injection over static, procedural code that is hard to understand & change. If you deal with scary legacy code you can use JMockit or Powermock to mock static methods.

Basically, if you’re writing new code, write it as non-static method and use dependency inject to inject singleton instance. And to test legacy static methods, use some other test framework that supports it. In my experience, I found JMockit and Mockito can co-exist in the same test code without a problem. Here’s the code to test static method using JMockit to stub the return value of a static method call:

new NonStrictExpectations() { SomeClass mockSomeClass; { mockSomeClass.aStaticMethod(__arguments__); result = __stubbed_result__; } };
3. Try to minimize dependencies

If you write unit tests first as in TDD, or as soon as possible (when you have some code that can compile and run, with placeholders for things not implemented yet), it will help you to identify clutter-ness of your code and make your code more modular. Modular code is easier to test as there are less dependencies to satisfy. It’s quite annoying when you have to mock or stub 10 interfaces/classes for the test setup because your implementation needs them all. It’s also code smell that your implementation is difficult for other people to use since they’ll face the same problem. If you are facing this situation, think hard about how you can make your code more modular so that some dependencies are moved or removed. If your method needs 10 external dependencies, chances are your method is growing too big and you should divide it into smaller logical pieces, each with less dependencies. That makes your code easier to read and understand as well.

That’s all for my first ever technology blog. Phew~~ I noticed that each section is shorter than the one before it, a sign of me getting more and more mentally exhausted. Blogging is hard!

P.S. Stubhub HIRING! We need talent. Come and join us!

Exclusive Behind the Scenes look at The Queens Club, London: Players Lounge

jackie tincher — Mon, 10 Jun 2013 21:23:08 +0000

This weeks sees the first of these partnerships, The Aegon Championships at The Queens Club taking place. The tournament is widely regarded as the men’s warm up tournament to Wimbledon and we’re pleased to share an exclusive glimpse behind the scenes at the Championships. Di Dougherty interviews the players’ concierge who gives us an insight into some of the requests he fulfills on behalf of the players competing this week.

Click here to view an exclusive behind the scenes tour of the Player Lounge.