Year 2020 starts on December 30, 2019. It’s true... I was not mistaken. So if your team has to deal with software where dates and timestamps are relevant, you might want to read this. We have been through this problem before and it was not easy to find out what was going on.

In the end of 2015, we were alerted to the importance of formatting dates, and the misuse of "YYYY" in some languages.

TL;DR:

In the beginning of 2017 the "BUG" attacked again, but in a more discreet way; so discreet that I took some days to notice what had happened. More than if we should use "YYYY" or "yyyy", the lesson to be retained is that as engineers we have to try to get to know the APIs, libraries, frameworks and such that we use. Know and test their operations, and be aware that even this may sometimes not be enough. And at the end of the day, always use "yyyy".

The story I will tell you happened in the backoffice (BO) of a resource management system. Fortunately, it wasn’t a critical error, but it could have been.

Chronicles of a "YYYY" in 2017

Once upon a time there was a user who ended the usage of a resource manually, for reasons beyond our control. The procedure in these cases is for the manager of the system to register the occurrence in the BO, to ensure that the timestamp is registered. However, for some reason, the BO was not allowing the manager to do so. Instead the system presented him with the message: "Enter a valid date".

The only restriction that exists in the code is that the timestamp for the end of activity must be later than the one for the beginning, and that it also should not exceed the current time. The user had started at 2017-01-01 22:07, and the manager was trying to close the timestamp at 2017-01-01 23:45. Since we were already on the 2nd of January, there was no doubt that the date entered was valid, and within the conditions.

We started by looking at the code, and we confirmed that the conditions were correct. We confirmed that there was no apparent reason for this error, and we felt we were facing a byzantine fault. As per usual, the team was under some pressure and we even briefly considered force-closing the session, so that the user would not be inconvenienced, while we tried to correct the problem. It would not be the first time.

This is a recurring situation with which you should also be familiar: the external pressure we are subject to, and that sometimes causes some errors to go by unnoticed, until they regrettably cause critical failures.

We debugged in production (PROD) and realized that the date the user was starting was being converted to 2016-01-01 23:45. And why? Exactly because we had put "YYYY" in the code instead of "yyyy".

For those who do not know, in a java.text.SimpleDateFormat (Java > 7), ‘Y’ and not ‘y’ represents what is called Week Year. That is, the year in which the week belongs to, and not the current year. As per the Week Year "convention", January 1, 2017 is still considered a week of 2016, and so in PROD the format() returns 2016 in the data field for the year. On the other hand, with "yyyy" it would return 2017.

But the story does not end here.

Do you think I would waste my time on this novella if it were just to remind you of the differences between a ‘Y’ and a ‘y’?

The problem is that "it worked on my machine". On all the developers’ machines that did this test using "YYYY", it printed 2017. That is, no one could replicate the bug. Or in other words, nobody could get the ‘Y’ to function as documented.

We started by solving the obvious and replaced the ‘Y’s with ‘y’s, and the system started to function as intended in all environments.

// some extra sarcasm {

Should we be concerned that ‘Y’ does not work locally? Would it be a problem? It could even be a desirable feature: we would never have to worry about choosing between lower and upper case!

}

We tried replicating the bug on different machines, and we were able to do it on the Pre-Production machine (PRE). This clearly increased the pressure for us to try to figure out what was going on. PRE is a local machine. The chances of this behavior being due to the misalignment of the stars with Amazon’s data-center had diminished.

As the engineers that we are, we could not let such inconsistencies go away. After all, we try to understand the reason for these problems, and we enjoy this search. We know that machines are always right, and we are aware that if we do not understand their motives well, sooner or later, we will be haunted by our lack of knowledge.

Why then did a ‘Y’ behave on our machines as if it were a ‘y’? In the java.text.SimpleDateFormat documentation we read:

If week year 'Y' is specified and the calendar doesn't support any week years, the calendar year ('y') is used instead. The support of week years can be tested with a call to getCalendar().isWeekDateSupported().

The isWeekDateSupported() from java.util.Calendar returns false because the Calendar is an abstract class that does not implement the Week Date concept, which is the basis of Week Year. However in the GregorianCalendar – which is the default implementation on this side of the globe – it returns true and the Week Date concept is in fact implemented. We have confirmed that in every environment isWeekDateSupported() returns true. It would therefore be expected that "YYYY" would give us 2016 in all environments… but no: in our machines, it continued to yield 2017. This was not where the problem came from.

(Note that, for example, if we ever stop by Thailand and/or our machines are set with their Locale there, our application and probably everything that belongs to package java.*, will eventually be using a BuddhistCalendar by default.)

However in this project we do not use java.util.Calendar. We use org.joda.time.DateTime from Joda, which even has a method that gives us the Week Year:

public int getWeekyear()

Get the weekyear field value.

The weekyear is the year that matches with the weekOfWeekyear field. In the standard ISO8601 week algorithm, the first week of the year is that in which at least 4 days are in the year. As a result of this definition, day 1 of the first week may be in the previous year. The weekyear allows you to query the effective year for that day.

This method confirmed to us that we were not crazy. In all environments (new DateTime(2017, 01, 01, 23, 45)).getWeekyear() returned 2016. We also learned that it was ISO-8601 that set the standard for the Week Year representation, at least in Joda DateTime.

We realized that if it was not a matter of the constitution of the date itself, it would have to be something to do with formatters. Out of curiosity and to test our sanity, we tried to do the same transformation, but this time using org.joda.time.format.DateTimeFormatter with "YYYY". We ended up getting 2017 in all environments, including PROD and PRE. But isn’t "YYYY" Week Year?

When we stopped practicing divination and went to read the org.joda.time.format.DateTimeFormat documentation, we realized that the DateTimeFormatter was working correctly, and we also began to realize that, in the end, conventions aren’t an exact science:

Yes, as you can see, the 'Y' in Joda does not mean Week Year... Well. So the problem was only with java.text.SimpleDateFormat. At least, the'y' continues to mean the same for everyone, and the rule "always use 'y'" continues to be valid.

We then entered that phase which we all love: where we no longer know where to start. We've forced timezones between other things that I'm too embarrassed to share. We even considered the chance of being the lucky ones with a bug derived from this year’s Leap Second.

Our last hope was that the developers had left a bug somewhere in that version of Java, and that it had already been fixed in the versions on our machines. Even if there were no record in any bug tracker, never lose hope! We then installed on one of our machines the same Java build as there was in PRE and PROD and… nothing. "YYYY" still did not give 2016 locally.

We then did what we should have done a long time ago. We went to explore the SimpleDateFormat code:

If up until now we thought that a SimpleDateFormat used only the location to translate symbols (e.g. Dec vs Dez, Mon vs Seg), then after all this we realized that it uses its own java.util.Calendar in the process of formatting dates, to where the Locale is passed.

The PROD and PRE machines were in en_GB, and the development machines where we had been testing locally were in en_US. Et voilà!

(At this point it started to make sense that in the SimpleDateFormat documentation there would be references to the Calendar, anyway…moving on!)

But wait: "So, if there’s a standard, such as ISO-8601, didn’t the United States adopt it?" Yes and no! Or in the middle, whichever you prefer.

To start with, it was clear to us that the java.util.Calendar is not ISO-80601 complaint, with regard to a Week Date: Java Calendar WEEK_OF_YEAR not ISO-8601compliant?

On the other hand, if you remember, when we previously tried to extract the Week Year from a DateTime, as it was org.joda.time.DateTime we always got 2016. And it was what we expected, that is, Joda respects the standard:

The standard considers the first day of the week as Monday. However, this is not true in the conventions of some countries such as the United States, which adopted the ISO-8601 for the representation of dates but diverge on the concept of Week. For instance, in the United States the first day of the week is Sunday. It is precisely this that, for good or bad, java.util.Calendar, more specifically the GregorianCalendar class, reflect.

So if you ever for some reason or chance really, really want to use the concept of Week Date and Week Year (with "YYYY" or in some other way), make sure your code does what you intend.

And if, in this sense, for some reason or chance, you really have to use a java.text.SimpleDateFormat and need to follow ISO-80601 standards, the best way to do it will be to calculate it manually. It is easy, as suggested by the "Leap Second" Wikipedia page, using the assumption that the first week is always the one containing the first Thursday of the month, as described in the standard.

Finally, to help the "party", in Java 8, we also have available the java.time.format.DateTimeFormatter class. Despite this new java.time.* package having the same Joda conventions, in this DateTimeFormatter, the ‘Y’ continues to mean Week Year, unlike Joda's DateTimeFormatter.

So, it's never too much to remember, always use 'y'... until the day when it no longer does what you want.

And on that day read the documentation of whatever you are using :)

Thirteen years ago, when we worked in an office the size of one of our current meeting rooms, I was the manager, developer, tester and devops engineer (at the time the term didn’t even exist) of one of our projects. I didn’t use any development methodology. I didn't even know what that was. The day another person started working on the project, the chaos began. Luckily, several people that came to work at Premium Minds introduced us to the Agile world.

Over the years, we have been experimenting and iterating until we got to our current model of Agile Coaching. This path of learning would not have been possible without the generosity of so many people in the world who share their experiences and ideas. In return, we are now sharing the document we currently hand to new Agile Coaches here at Premium Minds. This document explains our philosophy and the history behind it. We hope you find it useful. Questions and comments are welcome.

Introduction

The Agile Coach is a key element in our organizational structure. This document aims to clarify the role of an Agile Coach and outline a common work base.

What is the purpose of an Agile Coach?

The goal of an Agile Coach is to promote and care for team productivity and positivity. We believe that a productive and positive team is better prepared to overcome its challenges while giving room for professional and personal growth for each of its elements.

Productivity and positivity are concepts that cover a wide range of behaviors that can generally be understood as follows:

Productivity:

• Alignment of all team members towards their common purpose
• There are clear objectives and strategies
• There are clear and efficient decision-making processes
• There is leadership within the team, appropriate to each situation
• The team has all the resources and experience needed for their challenges
• There is responsibility for the commitments of each person and the team
• The team is proactive and approaches opportunities in a positive and creative way

Positivity:

• There is a safe space that promotes trust between people
• There is an environment of mutual respect and genuine positive regard
• There is a strong sense of belonging
• Clear and efficient communication is valued over “politicking”
• There is continuous feedback among colleagues, without criticism and defensiveness
• Different ideas, experiences and personalities are valued
• There are low levels of toxicity within communication

The Agile Coach is not expected to be responsible for this vast set of behaviors. The team itself should assume this responsibility, but it helps a lot to have an outside caretaker to promote these ideas and pay attention to areas where team members might have more difficulties.

It is desirable for the Agile Coach to be free from conflicts of interest with the different parts of the team. This exemption brings:

• A unique perspective of emerging issues
• The possibility to mediate and solve conflicts
• The ability to genuinely listen to each person, giving room for their vulnerabilities

The Agile Coach is inspired by the Agile philosophy and takes the Agile Manifesto into account.

What are the Agile Coach’s tools?

Being an Agile Coach at Premium Minds is a part-time job that takes up to 8 hours a week from someone who has their own team, their own challenges and their own Agile Coach :). This option has advantages and disadvantages that will be discussed in the chapter “Why Premium Minds has Agile Coaches”. However, we can already say that a direct effect of this option is that the Agile Coach is not always present with the team. For that reason, they use the O3 and team ceremonies as preferred opportunities to do their job.

At O3s and ceremonies, the Agile Coach uses mainly communication as the basic tool for their work. It is through communication that the Agile Coach helps each team member and the group to continually improve their productivity and positivity.

This means that the Agile Coach has to practice these three regularly:

• Empathy - the ability to put oneself in another's shoes without judgment, which helps understanding each person's context.
• Active Listening - listening without judgments creates the possibility of honest sharing of both successes and failures.
• Assertiveness - clarity in communicating helps reducing the normal entropy in relationships, stimulating focus on what is really important.

In addition to communication-related work, the Agile Coach can also contribute on:

• Designing and implementing processes in the tools used by the team: Jira, Trello, Confluence…
• Using productivity metrics such as: Sprint Burndown Chart, Cumulative Flow Diagram, etc…

This kind of technical contribution varies according to the needs of the team and the proficiency/motivation of the Agile Coach. It's OK not to do everything. It is also OK to be challenged to do more.

What are O3s?

The O3s are One on One conversations. These are regular conversations that the Agile Coach has with each team member at least once a month. These conversations are primarily intended to create a safe space for the other person to be able to talk about what they need to talk about, whether personal or professional.

These discussions are very important, as they are the main means of regularly following-up with people at Premium Minds. Without O3 meetings, a process of demotivating may not be detected in time for the situation to be taken care of. A process of demotivation is just one example of the important things that can become invisible without regular follow-up. Again, these conversations are very important.

The chat should begin with an open question. For example, “How are you feeling?”, “How are things going?”. The goal is not to condition the conversation in one direction but give room for the person to talk about what is really important to them. Many times the person will bring up problems for which they need a solution or response. It is important that the Agile Coach transforms this information into concrete steps to reach the solution. Most often the best is to hold the person responsible for finding it. The question, “What could you do?” is a good way to help the person think about the possibilities in hand. If it is important to talk to someone else, the Agile Coach should help make that dialogue happen. You can explore how this conversation would be like, or even suggest you be there to moderate the conversation. Sometimes the person will bring feedback about someone. The Agile Coach encourages feedback to be delivered directly. It is much more effective than being a proxy.

If the Agile Coach does not know what to do, they can ask for help from someone in the Premium Minds management team. There will be times when this is the only way, for example when the matter regards organizational issues between the person and the company. If the person does not want to talk, this is also OK, sometimes it may not be the ideal time or there might not be anything relevant to share. It is still important for the person to feel that the Agile Coach is available to listen to them. One of the concerns that usually arises is whether the Agile Coach should do an O3 with the Product/Project Manager. Yes, you should. The PM is also a member of the team and, if not included, the Agile Coach will have a poorer view of team dynamics and less chances of stepping in if needed. It’s also normal to have concerns.

An excellent reference for becoming an O3 ninja is the Manager Tools podcasts. You can start by listening to this one.

What are ceremonies?

Team ceremonies are moments in time that allow teams to stop, synchronize, and organize their work. Each team has a set of ceremonies inspired by the various methodologies that follow the Agile philosophy.

Here are some examples of ceremonies:

• Dailies - Brief daily meetings for work synchronization and obstacle identification, usually carried out while standing up.
          It's Not Just Standing Up: Patterns for Daily Standup Meetings
          Effective Standups around Kanban Board

• Plannings - Planning meetings held at the beginning of the sprint to negotiate the commitment to the next work block.

• Groomings - work unit estimation meetings.
          All you need to know about backlog grooming

• Retrospectives - Meetings to reflect on productivity and positivity within the team during the last period of time.
          http://www.funretrospectives.com/

• Reviews - Meetings with the whole team to present and celebrate the work done during the last period.

• Internal talks - Training meetings within the team where one or more people explain concepts, ideas, implementations, etc.

The ceremonies performed by each team are usually closely linked to the methodologies the team is using. These methodologies and their ceremonies should be chosen smartly and in agreement with the whole team, also with the understanding that projects are changeable and constantly evolving. It is not uncommon for a team to change their chosen methodology, for example, from Scrum to Kanban, because the project has entered a different phase. If we follow this principle borrowed from Kanban, we will always be on the right path:

"Start from the existing context while stimulating continuous, incremental and evolutionary changes to the system.”

Other tools

There are many other tools that can be used by Agile Coaches with which we have been experimenting over time here at Premium Minds. One example is the definition of OKRs, or the dynamics of toxins based on the work of the Gottman Institute. An even simpler approach is team lunches. Not wanting to limit or outline the means that the Agile Coach uses, we consider that the important thing is to experiment and share. The Agile Coaches forum and working groups are excellent opportunities to get to know old and new experiences. And this is part of the learning and evolution process of an Agile Coach.

How does an Agile Coach learn and evolve?

Besides this document there is no written documentation on the role of the Agile Coach at Premium Minds. In fact, we prioritize the evolutionary and iterative process of learning that happens between people over extensive documentation. We believe that the difficulties and problems that each Agile Coach faces within their team are, in fact, excellent opportunities for learning and evolving.

Thus, at Premium Minds, each Agile Coach belongs to a small group that works as a secure foundation for growth. This group usually consists of 3 Agile Coaches and 1 Facilitator. These groups, given their small size, are a privileged place to share difficulties, concerns, and problems. The group works as a support network, but it also enhances the personal and professional development of the Agile Coach. This means that an Agile Coach is responsible for their own evolution and for contributing to the evolution of other Agile Coaches.

In addition to these working groups, there is also the Agile Coaches forum where more ideas and experiences are discussed. In 2018, the forum approved the idea of ​​all Agile Coaches doing team coaching training, which happened during last year. Ideas like these are welcome.

Besides these internal resources to facilitate the work of an Agile Coach, there is a set of references that is always worth reading/seeing:

• http://labs.spotify.com/2014/01/07/how-it-is-to-be-an-agile-coach-intern-at-spotify/
• agile - Can Scrum and Project Management live together? - Stack Overflow
• From Scrum to Lean | Nettuts+
• https://blog.premium-minds.com

Why does Premium Minds have Agile Coaches?

Agile Coaches are key elements for understanding how Premium Minds is organized. When we became more than 30 people, we came across this slide from Netflix that made us think about the weight of management in our structure. As we continued to grow we aimed to follow a path of greater autonomy and responsibility of people and teams. This desire has been translated into the role of the Agile Coach as a champion of productivity and team positivity.

Over the years we have tried different implementations of this role: a Scrum Master that was part of the team, a Scrum Master for all the teams, etc. For some years now we have settled with a very particular configuration:

An Agile Coach for one or two teams, who is not an element of the team and has their own project, which makes them a part-time Agile Coach.

From our experience, this implementation has the following advantages:

• We have a wide and diverse forum of agile coaches, which brings a richer vision, with a great involvement from various parts of the organization.
• There is pollination among the teams (e.g. the Agile Coach from eos Payments belongs to the M&M team, whose Agile Coach belongs to the eos Parkcore team, and so forth), which positively impacts our company’s culture.
• Some people have the opportunity to experience a job with people management traits, which for many is an interesting addition to their technical work.

And the following disadvantages:

• The Agile Coach has to manage their time between two jobs. In times of crisis on one side, this management is very complicated.
• The Agile Coach is not present in the hustle of everyday life and can miss out on key moments of conflict or celebration in the team.

As with all Agile Coaches’ work, its implementation is also a process of continuous and incremental evolution, and therefore suggestions for improvement are always welcome.

Agile Coach Start Toolkit

Did you just accept the invitation to become an Agile Coach for one of our teams? If so, congratulations! This is a brief list of important actions you can take during the first few weeks of your new challenge. These actions will help you get a more effective and efficient start.

1. Get to know your team and their ceremonies (frequency, methodologies, etc.).
           a. If you are about to start working with a team that already had an Agile Coach, you can ask them to attend some of their ceremonies as a transition process.
2. Mark down all of the team’s appointments on your calendar: plannings, dailies, etc.
3. Join the team's chat channel.
4. Check out the existing mailing lists of the team and take ownership of them.
5. If you still haven’t, listen to this podcast: https://www.manager-tools.com/2005/07/the-single-most-effective-management-tool-part-1
6. Schedule an O3 with each element of your new team during the first week. Share who you are, what are your fears and hopes for this new role. In this first conversation, the more vulnerable you are the safer is the space you'll be creating for your relationship with the different team members.
7. Get together with your work group, introduce your team, and share your goals for group meetings, what you expect from them, and what they can expect from you.
8. Spend some time getting to know your team's board on JIRA (or Trello) so that when you have to change something you already have the know-how to do it.
9. Start filling the Team Canvas to better understand the team from different angles.
10. At the end of the week, log the hours you spent as an Agile Coach on the project of that team.
11. Seek to maintain an ongoing learning process about this new role. Some possibilities are:
            a. https://ryanripley.com/agile-for-humans/
            b. https://agilein3minut.es/
            c. https://itunes.apple.com/us/podcast/leadingagile-soundnotes/id995790407?mt=2
            d. https://www.manager-tools.com/2005/07/giving-effective-feedback#

Best of luck!

One of the top 10 most voted Visual Basic 6 questions on Stack Overflow is this one: Stop Visual Basic 6 from changing my casing.

It’s a problem that seems to affect a lot of people around the world. Well, not so much… after all, interest in this language is decreasing and it seems to be the most currently hated one. It makes sense, since Microsoft abandoned support for the IDE in 2008.

The version control system we use at Premium Minds is Git. For some time, we didn’t really pay any special attention to the capitalization in our VB6 application. We had implemented code review procedures, but the commit noise created by this IDE behavior caused a major high-impact bug to make it through the review unnoticed.

The solutions proposed on that Stack Overflow question never seemed practical to me: create a dummy class or hide code with compiler directives.

Our solution is the adoption of a tool and some team discipline.

The tool is a script that compares the files on disk with the ones from the previous commit, creates a diff that ignores case changes and applies the resulting patch.

The team discipline is simply to invoke this script before each commit and reject any code review where the script did not run.

The content of the script is as follows:

cat git_ignore_case.sh 

#!/bin/bash

set -e

#forms, classes and modules
for file in $(git status --porcelain | grep -E "^.{1}M" | grep -E -v "^R" | cut -c 4-| grep  -e "\.frm" -e "\.bas" -e "\.cls" -e "\.Dsr"); do
	ORIGFILE=$(mktemp)
	PATCHFILE=$(mktemp)
	git cat-file -p :$file > $ORIGFILE
	diff -i --strip-trailing-cr $ORIGFILE $file > $PATCHFILE || true
	patch -s $ORIGFILE < $PATCHFILE
	cp  $ORIGFILE $file
	rm  $ORIGFILE $PATCHFILE 
	unix2dos --quiet $file
done

#projects
for file in $(git status --porcelain | cut -c 4-| grep -e "\.vbp$"); do
	echo $file
	ORIGFILE=$(mktemp)
	PATCHFILE=$(mktemp)
	PROCESSEDFILE=$(mktemp)
	echo $file $ORIGFILE $PATCHFILE $PROCESSEDFILE
	git cat-file -p :$file > $ORIGFILE
	cat $ORIGFILE | cut -d "#" -f 1-3 > $PROCESSEDFILE
	diff -i --strip-trailing-cr $PROCESSEDFILE <(cat $file| cut -d "#" -f 1-3) > $PATCHFILE || true
	patch -s $ORIGFILE < $PATCHFILE
	cp $ORIGFILE $file
	unix2dos --quiet $file
done

To illustrate the problem, imagine this event handler for button Command1:

Private Sub Command1_Click()
    Dim str As String
    str = "Hello World!"
    MsgBox str
End Sub

Let’s add the files to git and commit:

git add Form1.frm Project1.vbp
git commit -m "first commit"

Now, we add a Command2 button with the following event handler:

Private Sub Command2_Click()
    Dim Str As String
    Str = "Hello World!"
    MsgBox Str
End Sub

Note that we used Str instead of str, this time. If we look at the differences at this time, we have:

git diff Form1.frm
diff --git a/Form1.frm b/Form1.frm
index a77250e..f8563ce 100644
--- a/Form1.frm
+++ b/Form1.frm
@@ -44,7 +44,14 @@ Attribute VB_Exposed = False
 Option Explicit
 
 Private Sub Command1_Click()
-    Dim str As String
-    str = "Hello World!"
-    MsgBox str
+    Dim Str As String
+    Str = "Hello World!"
+    MsgBox Str
 End Sub
+
+Private Sub Command2_Click()
+    Dim Str As String
+    Str = "Hello World!"
+    MsgBox Str
+End Sub
+

The VB6 IDE altered Command1_Click, which was not our intention, and as far as it is known, there is no option to turn this behavior off.

If we apply our script:

./git_ignore_case.sh

Evaluating our differences again:

git diff Form1.frm
diff --git a/Form1.frm b/Form1.frm
index a77250e..5abbb5d 100644
--- a/Form1.frm
+++ b/Form1.frm
@@ -48,3 +48,10 @@ Private Sub Command1_Click()
     str = "Hello World!"
     MsgBox str
 End Sub
+
+Private Sub Command2_Click()
+    Dim Str As String
+    Str = "Hello World!"
+    MsgBox Str
+End Sub
+

We have only the changes we performed.

The script is not infalible, there are some scenarios where it fails, but it is the best solution to this problem I know of to this day.

Commits are cleaner, code reviews are smoother and the bugs caused by this IDE behavior have decreased or disappeared altogether.

An agile team is closely tied to what I consider to be the core values of Premium Minds: the quality of the end product, the life quality of the people working on the product, and, of course, the Agile Manifesto.

After two and a half years working as an Agile Coach for several diverse teams, I have decided to write some considerations about what, as I see, contributes to building an agile team.

1. Don’t focus on the approach

All methodologies are welcome: Kanban, Scrum, Lean, Extreme Programing… What matters is choosing a methodology to which the team is able to adapt well and vice versa. Also, we must not forget to include the client in the decision process.

Some changes to Scrum that are common in our teams are, for example, the duration of sprints (one week, not two), the presence of the project manager at dailies, a longer period between retrospective meetings and not at every sprint, etc.

2. Tools are important

This may seem contradictory in regards to the Agile Manifesto, but being “important” doesn’t mean they are more relevant than the people and their interactions.

Tools like Trello or Jira are extremely useful to keep some order in the development process of a team. For that, it’s necessary to know how to work with them and leverage their potential. It doesn’t make sense to create a daily burnChart in paper if Jira can provide you with that information.

In short, using the available tools is important and, many times, due to some strict readings of the Manifesto, it can be forgotten.

3. Team mindset

Having a clear team mindset that is common to everyone is the first step to having a united group more easily open to change. Keeping a team aligned with the process, the client, and the mindset can be a lengthy and tiring procedure that should be adjusted several times.

Some examples of good practices for the alignment of this mindset are one-on-one meetings - which should be done at least once a month with each team member - retrospectives and constant feedback exchange. This policy of conversation and honesty makes it easier for people to be aligned and therefore to reach a common purpose.
_

We’ve already talked about methodologies, tools, and team management, so now let’s move on to the part that can be overlooked when talking about Agile: code development. The next two items will address it.

4. Good tests != Lots of tests

An important part of creating durable and high quality software is being able to perform good tests. Tests are a well debated theme in the realm of Agile. In the case of Extreme Programming, the concept of “test driven development” is used; in some adaptations of Kanban, tests are considered part of the development cycle; but in the everyday life of a team, tests are often left behind due to deadlines or bad setups.

Tests can be divided in 2 types:

• Unit testing - According to the book “Pragmatic Unit Testing in Java with JUnit” (even though its focus on Java, it contains very broad characteristics), a good unit test must be “ATRIP”:

  • Automatic;
  • Thorough;
  • Repeatable;
  • Independent;
  • Professional / Readable;
  • Fast - This feature is not part of the acronym but is also important. The tests should run fast for easier adaptation to problems.

• Integration testing - Not my favorite, but still I want to highlight some important points:

  • Test cases should be separated for a better understanding of the problems;
  • A good practice is the detailed documentation of both test cases and test results;
  • They must focus on functionalities rather than business logic;
  • They must be run at specific stages of the development process, due to the excessive time they take.

5. Refactoring

No one writes the best code in the world at first try, and even if you could, it can be very volatile and all code improves when refactored. Every developer understands the importance of a refactor, but many times they look at it as if it were a seven-headed beast.

There are several refactor techniques and, in my perspective, not all refactors have to be long. The advice I give to the teams I’ve worked with is always to approach the problems in small bits, little by little. For this to happen in a lighter way, you need to have a good test base to be able to do minor refactorings. This method makes the code better and more readable every time you do a refactoring, so it becomes a better code in the long run. The time allocated for this is usually negotiated with the client and shows the resulting added value from the process.

6. More automations = More development time

A good title for this point 6 would be "Continuous Integration", but I want to cover a few more aspects. Having only a seamless integration environment does not provide a team everything that I think is needed to be more agile when it comes to developing, but it's the beginning of it all.

In my perspective, a project with agile continuous integration should have the following characteristics (if applicable):

• Automatic database migrations, especially schema and mandatory data migrations;
• Configurable and easy-to-access data migration system: providing an easy and accessible way for everyone on the team to insert data that was already testes (unitarily) in an integration environment;
• Running tests automatically in a common environment;
• Creating the deploy package in a common and controlled environment;
• Button Deploy: offering an easy way to deploy the latest version of the code that is accessible to the whole team;
• Post-deployment automation: how to run the integration tests.

Of course, for all this to happen, we must have many other changes in the development phase in order to ensure, for example, the idempotency of data migrations. Another aspect that helps in these cases is having a system that can do the builds and deploys, such as Jenkins or Travis CI.

7. Time

Nothing happens overnight, and if there is one thing that I think I’m faulting as an Agile Coach is seeking more quick results than long-term results. There is nothing wrong with looking for "quick wins", but an agile team has to be built in the long run. Refining a team is a time consuming process. It’s not just a single session that will have the power to align the expectation of a team, nor the whole structure of continuous integration will be done within a week. It may never even happen.

In conclusion, these are the seven points that I think are most important in building the foundation of an agile team nowadays.

There’s a SQL query idiom I see a lot, which I’d like to discuss. Using the classic Northwind database example, let’s look at the two tables Shippers and Orders:

CREATE TABLE Shippers(
	ShipperID int IDENTITY(1,1) NOT NULL,
	CompanyName nvarchar(40) NOT NULL,
	Phone nvarchar(24) NULL,
	PRIMARY KEY (ShipperID)
)

CREATE TABLE Orders(
	OrderID int IDENTITY(1,1) NOT NULL,
	CustomerID nchar(5) NULL,
	EmployeeID int NULL,
	OrderDate datetime NULL,
	RequiredDate datetime NULL,
	ShippedDate datetime NULL,
	ShipVia int NULL,
	Freight money NULL,
	ShipName nvarchar(40) NULL,
	ShipAddress nvarchar(60) NULL,
	ShipCity nvarchar(15) NULL,
	ShipRegion nvarchar(15) NULL,
	ShipPostalCode nvarchar(10) NULL,
	ShipCountry nvarchar(15) NULL,
	PRIMARY KEY (OrderID ),
	FOREIGN KEY (ShipVia) REFERENCES Shippers(ShipperID)
) 

If we want a listing of the number of orders for each shipper, w3schools.com tells us we can use:

SELECT Shippers.CompanyName, COUNT(Orders.OrderID) AS NumberOfOrders 
	FROM Orders
	LEFT JOIN Shippers ON Orders.ShipVia = Shippers.ShipperID
	GROUP BY CompanyName;

But the GROUP BY is done by name. If two shippers exist with the same name, they’ll be grouped in the same line, rather than listed separately. The query could also be done with a RIGHT JOIN to include suppliers with no orders.

The query should group by ShipperID, which is the primary key:

SELECT Shippers.CompanyName, COUNT(Orders.OrderID) AS NumberOfOrders 
	FROM Orders
	LEFT JOIN Shippers ON Orders.ShipVia = Shippers.ShipperID
	GROUP BY ShipperID;

Alas, this query isn’t valid. SQL Server will complain with “Column 'Shippers.CompanyName' is invalid in the select list because it is not contained in either an aggregate function or the GROUP BY clause.”

As the error states, CompanyName cannot be used like that. We can always replace CompanyName by ShipperID in the SELECT clause, like we did in the GROUP BY clause.

SELECT Shippers.ShipperID, COUNT(Orders.OrderID) AS NumberOfOrders 
	FROM Orders
	LEFT JOIN Shippers ON Orders.ShipVia = Shippers.ShipperID
	GROUP BY ShipperID;

But the fussy manager we’re turning the query results in to doesn’t care about each shipper’s database identifier. No, we actually need to include the name.

One of the most frequently suggested solutions is the use of an aggregation functions, even though we know it is not needed, as there is only one CompanyName for each ShipperID. This leaves us with:

SELECT MAX(Shippers.CompanyName), COUNT(Orders.OrderID) AS NumberOfOrders 
	FROM Orders
	LEFT JOIN Shippers ON Orders.ShipVia = Shippers.ShipperID
	GROUP BY ShipperID;

Here, we used MAX(), but we could also have used MIN(), or even AVG() if the field were numerical. Ultimately, we might even have used a random aggregation function. This ambiguity is what turns me away from this solution.

There is a solution involving funcional dependencies. If the database supports these, it will realize that CompanyName depends on ShipperID and thus the second query that resulted in an error would now work. PostgreSQL supports this from version 9.1 onward. But I’m working with SQL Server so this is no good for me.

MySQL is famous for not reporting errors and returning random values, when using a column in the SELECT clause that is not in the GROUP BY clause, but since version 5.7.5, it respects functional dependencies.

Another frequently suggested solution is to add the CompanyName to the GROUP BY:

SELECT Shippers.CompanyName, COUNT(Orders.OrderID) AS NumberOfOrders 
	FROM Orders
	LEFT JOIN Shippers ON Orders.ShipVia = Shippers.ShipperID
	GROUP BY ShipperID, Shippers.CompanyName;

This works, but in a more complex query dealing with more tables it becomes hard to understand which GROUP BY columns are actually needed for grouping and which are there just to appease the database gods.

By the same token, if we want to extract an addtional field, like say the Phone, we have to add it to the SELECT clause and the GROUP BY clause:

SELECT Shippers.CompanyName, Shippers.Phone, COUNT(Orders.OrderID) AS NumberOfOrders 
	FROM Orders
	LEFT JOIN Shippers ON Orders.ShipVia = Shippers.ShipperID
	GROUP BY ShipperID, Shippers.CompanyName, Shippers.Phone;

This seems like the wrong approach. I’ve recently had to add a couple of columns to several similarly structured queries and it felt bad to contaminate the GROUP BY with additional unnecessary columns.

If the problem is mixing the CompanyName column in the SELECT and the GROUP BY, it can be solved by separating the GROUP BY into a subquery with the desired aggregations and do whatever JOINs are necessary to obtain the desired details.

SELECT Shippers.CompanyName, NumberOfOrders 
	FROM (
			SELECT ShipVia, COUNT(Orders.OrderID) AS NumberOfOrders 
			FROM Orders
			GROUP BY ShipVia
	) AS OrdersSummary
	LEFT JOIN Shippers ON OrdersSummary.ShipVia = Shippers.ShipperID

This way, if we want to add the Phone, we just need to do it in the SELECT clause:

SELECT Shippers.CompanyName, Shippers.Phone, NumberOfOrders 
	FROM (
			SELECT ShipVia, COUNT(Orders.OrderID) AS NumberOfOrders 
			FROM Orders
			GROUP BY ShipVia
	) AS OrdersSummary
	LEFT JOIN Shippers ON OrdersSummary.ShipVia = Shippers.ShipperID

My goal was to discuss several possible approaches to GROUP BY queries and highlight one I don’t see recommended often enough on search engines or sites like stackoverflow, among others.

*bica is a typical portuguese expression for a small cup of coffee.

It all began with a need to drink coffee! Over spring, every day, after having lunch in the galley, there was a widespread urge to drink coffee… without having to climb all the way up to the 3rd floor, where the coffee machine was in our old office.

Through some volunteering, or lot drawing, process, someone would climb the steps, bring back the coffee and happiness would ensue.

The first invention took the form of the "coffee-copter", a cut-up cardboard box allowing the transportation of up to 6 coffees. The second part of the invention never actually took place: a notepad for the "volunteer" to write up the orders and not forget whether each coffee was with or without sugar. By the end of spring the system hand’t changed much, but the habit remained.

Eventually, during one of the lunches, someone blurted out: "It would be awesome if I could call up a cup of coffee without having to get off my chair." We had just migrated to Slack and there was a lot of eagerness to fiddle with the integrations.

The idea took off: we decided to build a first version during creative day. The goal: to build a functional version of BICA in just one day.

On the day we launched BICA we didn’t say anything to the team! The first requests were born of curiosity and the simple "UX", a tablet placed near the coffee machine. This was one of the funniest parts of the project.

Social Effect

From the beginning, we imagined that BICA would have an amusing social effect. When you request a cup of coffee from BICA, you don’t know if it’s going to come and/or who is going to bring it. When it arrives, there is a fleeting moment of gratitude and happiness.

One of the most memorable moments starred Leo: he was stuck on a Skype call he couldn’t leave, praying for his first morning coffee. When he no longer believed it would come, Sandro slips into the room with a brain (short espresso, no sugar, of course!) and, discreetly, hands him the magic drink.

Architecture

BICA consists of three simple parts:

● Slack Command: /bica used by ordinary mortals to request their coffee from their computer or mobile device (wherever Slack is running);
● Server: a webservice that receives requests from Slack and serves a simple web page showing active requests;
● Tablet: a screen placed by the coffee machine displaying the active requests page.

Slack Command
Slack allows us to create custom commands, which can call a REST endpoint.
When someone types "/bica [something]" Slack makes a POST to the endpoint with a well known json content.

/bica coffee no sugar

This Slack command triggers the following request at the endpoint:

 POST https://bica.pminds.com/coffeeRequest
 {
    username: "diogo.figueiredo",
    message: "coffee no sugar",
    ...
 }

Simple. Next step: create a server service to parse this json and create coffee requests.

Server
We built a simple Scala server (Play Framework). We defined the "/coffeeRequest" endpoint that accepts the POST and JSON body.

def coffeeRequest(body: Request) -> {
    def arguments = body.fromJson()
    def request = memory.save(arguments.username, arguments.message)
 
   sendNotification(request.callbackUrl)

   Ok()
 }

Part of the Slack request includes a callbackUrl that we can call to give feedback to the user. We used this to notify the user that (1) the request was accepted, (2) the coffee is on its way (further down the line).

Tablet
Finally, we had to decide how the requests would interact with everyone else. We knew we didn’t want spam (if the idea was to ask for coffee on the #general channel, BICA need not have been built).

We felt (as opposed to knowing) that, if it was to work, the process could not be complicated. We designed various wireframes and voilá! - a tablet placed by the coffee machine. Thus, whoever went to get coffee for themselves could take the opportunity to make someone happier by taking him a cup of coffee requested via BICA.

Metrics

We decided from day one that it would be very interesting (and somewhat important) to have metrics, to understand if BICA was being used, when, and by whom.
We defined two simple events: the coffee request (along with who made it) and when are the requests being served. This allows us to know that last month 90 coffees were requested, 85% of which were delivered!

Other notes

For those of you that like technical details, we leave you with this appendix.

● On the Server, we used the Play Framework because it comes with built in easy REST JSON request handling and simple HTML page serving;
● On Slack, we decided that people could write what they wanted on the message. This message appears as-is on the tablet and that made things very simple (there is no need for the user to select among options and there is no need to build a parser or other complicated bits);
● On the Tablet, we ended up writing up a simple Android app: a webview that loads BICA in kioskmode. Before we did that, someone would fiddle with the tablet once in a while and leave it running Youtube :-)

I want BICA too!

Of course! We’re open-sourcing BICA and creating an INSTALL guide. If you’d like to be notified when that happens, send us an email to bica@pminds.pt.

There is a burst of energy that happens upon finding the source of an elusive bug, which is driving me to write this post.

On a recent project, we used the Apache Wicket framework for the web interface. As the client also wanted a very simple REST API, we used the Wicket REST annotations library from the Wicket Stuff project.

The project had an abstract base class ResourceBase, inherited by all endpoints:

public abstract class ResourceBase<T extends IDTOFrontEnd> extendsAbstractRestResource<JsonWebSerialDeserial> {	
    public ResourceBase(JsonWebSerialDeserial serialDeserial) {	
        super(serialDeserial);
    }
    
    public abstract T submit();
}

Here’s one of the endpoints:

public class LoginResource extends ResourceBase<IRestLoginDTO> {
	public LoginResource(JsonWebSerialDeserial serialDeserial) {
		super(serialDeserial);
	}

	@Override
	@MethodMapping(value = "/")
	public IRestLoginDTO submit() {	
		return null;
	}
}

The IRestLoginDTO interface extends IDTOFrontEnd:

public interface IDTOFrontEnd extends IClusterable {
}

public interface IRestLoginDTO extends IDTOFrontEnd {
}

Everything worked OK in development, but when we deployed in pre-production, we suddenly got the following error when accessing the API:

Ambiguous methods mapped for the current request: URL 'api/common/login', HTTP method GET. Mapped methods: submit

Everything worked well in one of my colleagues’ machine, but I was getting mixed results trying to reproduce the problem: sometimes it would work, and other times not, and I couldn’t figure out why.

As I had recently squandered a great deal of time on an OpenJDK bug that did not exist on Oracle JDK, I thought it might be the same type of thing, but I didn’t know what exactly was happening.

The error stemmed from a Wicket Stuff internal algorithm that tries to find the method which best matches a requested endpoint. When it finds two methods that match equally well, it throws an exception.

Here’s some code that illustrates the problem:

public class Demo {
	public static void main(String[] args) {
		Method[] declaredMethods = LoginResource.class.getDeclaredMethods();
		for (Method method : declaredMethods) {
			System.err.println(method.getName());
			System.err.println(method.getAnnotation(MethodMapping.class));
			System.err.println("--------------------");
		}
	}
}

This generates the following output:

submit
@org.wicketstuff.rest.annotations.MethodMapping(produces=application/json, httpMethod=GET, consumes=application/json, value=/)
--------------------
submit
null
--------------------

The algorithm used by Wicket Stuff ignores the methods that didn’t have the @MethodMapping annotation. The result above is the output on a machine where the application worked. On machines where it didn’t, the second method would also list the annotation, which leads to the method matching ambiguity that Wicket Stuff complains about.

The first question is, why is getDeclaredMethods() returning two methods? Is it because we have a submit method in LoginResource overriding the submit method in ResourceBase? The documentation tells us that getDeclaredMethods() excludes such overrides:

“Returns an array containing Method objects reflecting all the declared methods of the class or interface represented by this Class object, including public, protected, default (package) access, and private methods, but excluding inherited methods.”

If we alter the Demo class to also report the declaring class for each method, we have:

public class Demo {
	public static void main(String[] args) {
		Method[] declaredMethods = LoginResource.class.getDeclaredMethods();
		for (Method method : declaredMethods) {
			System.err.println(method.getName());
			System.err.println(method.getDeclaringClass().getName());
			System.err.println(method.getAnnotation(MethodMapping.class));
			System.err.println("--------------------");
		}
	}
}

And the output:

submit
LoginResource
@org.wicketstuff.rest.annotations.MethodMapping(produces=application/json, httpMethod=GET, consumes=application/json, value=/)
--------------------
submit
LoginResource
null
--------------------

The LoginResource class appears twice and the ResourceBase class does not appear at all. So, it does not appear to be related to inheritance. Or is it? We can alter both classes such that inheritance is no longer in the picture:

public abstract class ResourceBase<T extends IDTOFrontEnd> extends AbstractRestResource<JsonWebSerialDeserial> {
	public ResourceBase(JsonWebSerialDeserial serialDeserial) {
		super(serialDeserial);
	}
}

public class LoginResource extends ResourceBase<IRestLoginDTO> {
	public LoginResource(JsonWebSerialDeserial serialDeserial) {
		super(serialDeserial);
	}
	@MethodMapping(value = "/")
	public IRestLoginDTO submit() {
		return null;
	}
}

Running the Demo again:

submit
LoginResource
@org.wicketstuff.rest.annotations.MethodMapping(produces=application/json, httpMethod=GET, consumes=application/json, value=/)
--------------------

Humm, how about that? It turns out that inheritance does seem to affect the problem. Could it be related to the use of generics? We removed the generic type declaration on ResourceBase to test:

public abstract class ResourceBase extends AbstractRestResource<JsonWebSerialDeserial> {
	public ResourceBase(JsonWebSerialDeserial serialDeserial) {
		super(serialDeserial);
	}

	public abstract IDTOFrontEnd submit();
}

public class LoginResource extends ResourceBase {
	public LoginResource(JsonWebSerialDeserial serialDeserial) {
		super(serialDeserial);
	}

	@Override
	@MethodMapping(value = "/")
	public IRestLoginDTO submit() {
		return null;
	}
}

Running the Demo class again:

submit
LoginResource
@org.wicketstuff.rest.annotations.MethodMapping(produces=application/json, httpMethod=GET, consumes=application/json, value=/)
--------------------
submit
LoginResource
null
--------------------

So, the use of generics is not the problem. Do the IDTOFrontEnd and IRestLoginDTO interfaces have any influence on this? Changing LoginResource to:

public class LoginResource extends ResourceBase {
	public LoginResource(JsonWebSerialDeserial serialDeserial) {
		super(serialDeserial);
	}

	@Override
	@MethodMapping(value = "/")
	public IDTOFrontEnd submit() {
		return null;
	}
}

Running the Demo, we have a single result again:

submit
LoginResource
@org.wicketstuff.rest.annotations.MethodMapping(produces=application/json, httpMethod=GET, consumes=application/json, value=/)
--------------------

So, at this time, we know that inheritance is not supposed to be a problem but it is, and that the type of the return function is important.

I vaguely recalled reading about java’s bizarre behavior regarding equal methods with different return types, so I did what experts do and turned to javap:

javap LoginResource.class

public class LoginResource extends ResourceBase<IRestLoginDTO> {
  public LoginResource(org.wicketstuff.rest.contenthandling.json.webserialdeserial.JsonWebSerialDeserial);
  public IRestLoginDTO submit();
  public IDTOFrontEnd submit();
}

It checks out: both methods exist with different return types. But why? And where is the @MethodMapping annotation, important for Wicket Stuff? We need the -verbose javap flag to see annotations:

javap -v LoginResource.class

Classfile LoginResource.class
  Last modified 11/dez/2018; size 1067 bytes
  MD5 checksum b0e8a2040ab48404657b894a4954f35a
  Compiled from "LoginResource.java"
public class LoginResource extends ResourceBase<IRestLoginDTO>
  minor version: 0
  major version: 52
  flags: ACC_PUBLIC, ACC_SUPER
Constant pool:
   #1 = Class              #2             // LoginResource
   #2 = Utf8               LoginResource
   #3 = Class              #4             // ResourceBase
   #4 = Utf8               ResourceBase
   #5 = Utf8               <init>
   #6 = Utf8               (Lorg/wicketstuff/rest/contenthandling/json/webserialdeserial/JsonWebSerialDeserial;)V
   #7 = Utf8               Code
   #8 = Methodref          #3.#9          // ResourceBase."<init>":(Lorg/wicketstuff/rest/contenthandling/json/webserialdeserial/JsonWebSerialDeserial;)V
   #9 = NameAndType        #5:#6          // "<init>":(Lorg/wicketstuff/rest/contenthandling/json/webserialdeserial/JsonWebSerialDeserial;)V
  #10 = Utf8               LineNumberTable
  #11 = Utf8               LocalVariableTable
  #12 = Utf8               this
  #13 = Utf8               LLoginResource;
  #14 = Utf8               serialDeserial
  #15 = Utf8               Lorg/wicketstuff/rest/contenthandling/json/webserialdeserial/JsonWebSerialDeserial;
  #16 = Utf8               submit
  #17 = Utf8               ()LIRestLoginDTO;
  #18 = Utf8               RuntimeVisibleAnnotations
  #19 = Utf8               Lorg/wicketstuff/rest/annotations/MethodMapping;
  #20 = Utf8               value
  #21 = Utf8               /
  #22 = Utf8               ()LIDTOFrontEnd;
  #23 = Methodref          #1.#24         // LoginResource.submit:()LIRestLoginDTO;
  #24 = NameAndType        #16:#17        // submit:()LIRestLoginDTO;
  #25 = Utf8               SourceFile
  #26 = Utf8               LoginResource.java
  #27 = Utf8               Signature
  #28 = Utf8               LResourceBase<LIRestLoginDTO;>;
{
  public LoginResource(org.wicketstuff.rest.contenthandling.json.webserialdeserial.JsonWebSerialDeserial);
    descriptor: (Lorg/wicketstuff/rest/contenthandling/json/webserialdeserial/JsonWebSerialDeserial;)V
    flags: ACC_PUBLIC
    Code:
      stack=2, locals=2, args_size=2
         0: aload_0
         1: aload_1
         2: invokespecial #8                  // Method ResourceBase."<init>":(Lorg/wicketstuff/rest/contenthandling/json/webserialdeserial/JsonWebSerialDeserial;)V
         5: return
      LineNumberTable:
        line 12: 0
        line 13: 5
      LocalVariableTable:
        Start  Length  Slot  Name   Signature
            0       6     0  this   LLoginResource;
            0       6     1 serialDeserial   Lorg/wicketstuff/rest/contenthandling/json/webserialdeserial/JsonWebSerialDeserial;

  public IRestLoginDTO submit();
    descriptor: ()LIRestLoginDTO;
    flags: ACC_PUBLIC
    RuntimeVisibleAnnotations:
      0: #19(#20=s#21)
    Code:
      stack=1, locals=1, args_size=1
         0: aconst_null
         1: areturn
      LineNumberTable:
        line 19: 0
      LocalVariableTable:
        Start  Length  Slot  Name   Signature
            0       2     0  this   LLoginResource;

  public IDTOFrontEnd submit();
    descriptor: ()LIDTOFrontEnd;
    flags: ACC_PUBLIC, ACC_BRIDGE, ACC_SYNTHETIC
    Code:
      stack=1, locals=1, args_size=1
         0: aload_0
         1: invokevirtual #23                 // Method submit:()LIRestLoginDTO;
         4: areturn
      LineNumberTable:
        line 1: 0
      LocalVariableTable:
        Start  Length  Slot  Name   Signature
}
SourceFile: "LoginResource.java"
Signature: #28                          // LResourceBase<LIRestLoginDTO;>;

The annotation shows up in the Constant pool with id #19. That same id appears in the RuntimeVisibleAnnotations section of method IRestLoginDTO.submit(). Method IDTOFrontEnd.submit() has no annotation section.

I could not yet explain why the compiler had created the second method, but the ACC_BRIDGE and ACC_SYNTHETIC flags looked interesting. Researching these flags is how I learned about synthetic methods and bridge methods. That link explains that the compiler creates additional methods with the necessary signatures so it can deal with type erasure.

It is also created when a class extends another class and the return type of the overriding method is a subclass of the return type of the overridden method.

Now we can finally explain why getDeclaredMethods() returns two entries.

What about the annotation, though? Why will this sometimes work on my machine, and other times not?

Still suspecting a compiler bug, after much searching, I found bug JDK-6695379 : Copy method annotations and parameter annotations to synthetic bridge methods. But, this bug was fixed in version 8b94, since 2015, and I’m in 2018 with version 8u191.

Had the bug returned?

Around that time, I realized that the code didn’t work when compiled at the command line, but it worked when compiled from within Eclipse. My perplexity remained...

Why would .class files created by Eclipse be different from the ones created with the command line using either OpenJDK or Oracle JDK?

Eclipse has its own implementation of Git and its own implementation of Maven, could it also have its own compiler? Yep, Eclipse has its own compiler: JDT Core (also known as ECJ).

Does this compiler still suffer from the bug, which has been fixed on OpenJDK for several years? Yes, it does and the bug is reported: Bug 495396 - Inherited annotations are not visible on bridge methods (JDK 6695379).

Finally, everything fits! This explains why it worked when compiled with Eclipse, which does not copy the annotation to the synthetic method created by the compiler to deal with type erasure; but not when compiled with OpenJDK, which does copy the @MethodMapping annotation to the synthetic method and confuses Wicket Stuff.

A bug and subsequent pull request were reported on the Wicket Stuff project, to deal with synthetic methods correctly.

It’s not every day that 3 bugs (Wicket Stuff, OpenJDK e JDT CORE) conspire against you.

Image source: https://twitter.com/msdev/status/511877167832371200

Our Product Manager Riddhi Jamnadas was chosen to be the first guest on #GirlsWhoProduct, by Productized. We are proud to republish the interview she gave to Katsiaryna Drozhzha (who was also in charge of the photos).
We hope you enjoy to know Riddhi and her thoughts about being a woman in product Management.

The first girl who products featured in this project is Riddhi Jamnadas. Riddhi is a Product Manager at Premium Minds, a software house based in Lisbon, Portugal. After 8 years of experience in Marketing for IT companies, Riddhi has successfully transitioned into Product Management. In her spare time, Riddhi thinks about new product ideas, traveling around Europe, volunteering for social causes that can ‘make difference in people’s life’ and jogging with her Ipod Shuffle — because ‘no music, no life’

How did you make the move from being in marketing to becoming a Product Manager?

I come from a marketing background, so before getting into Product Management I’ve already had this product mindset, even though back than I still didn’t know what exactly Product Management was and where I could learn about it. Luckily, a friend of mine who was working in the field, told me about a great company that happened to be hiring a Product Manager with marketing skills. She helped me understand the responsibilities of that role and gave some practical advice, which I found to be very helpful. So finally I thought, “Why not? This field could be right for me! My background is marketing, but my field of studies is management, both of these experiences could help me complete the role.’’ I wanted to try something new and that was the reason why I’ve accepted the challenge.

Every year we see more and more gender diversity in the product community. However, women account for a minority of the leading product roles. Why do you think women are still grossly underrepresented in the product world?

I think that lack of information is the main stumbling block on the way to aspire more women to apply to product management jobs. Women usually don’t know what the product management is and what are the responsibilities or tasks in field that could complete their role, because this job is usually by mistake being related to technology industry experience. However, I don’t see it as a problem. Product Management is something relatively new, something that came with the evolution of the digital. Since it’s just the beginning, there is still room for improvement. Product world is constantly growing and we’ll definitely need more women to get the product jobs done.

What are the challenges we face in closing the gender gap in product world?

The main challenge that we have nowadays is how to inform women that Product Management is NOT necessarily coming from IT field and that a non-technical product manager can also be successful. Yes, it’s true that in the beginning more opportunities in product management were coming from the IT and engineering areas, mainly because it was necessary to have someone to build the product and the roadmap of the product. Nowadays, we have lots of alternative ways that help girls know more about the field. There are many possibilities out there, but of course if you have a non-tech background, you will not try to grow in the field that you think is designed for software engineers. Actually, that was my case as well. First I thought, “Maybe this job is for engineers and I shouldn’t try it.” Only later on, already working as a Product Manager, I’ve discovered that my role was more about planning, having management skills, the product roadmap and strategy prepared for the team and understanding the market needs,- these are the skills that can help to make the product better.

In spite of being a woman, I have never felt any difference

Do you think that women have distinctive skills that make a perfect fit for a product management job?

Honestly, I don’t think that there is a difference between men and women in performing the role in this field. I think our skills are the most important asset we have. In spite of being a woman, I have never felt any difference between me and my male colleagues. The only thing that I have needed to understand is not whether men and women have different working styles, but how to have someone senior to help me develop the skills that I need. Not because I am a woman, but because I am in this field for a short time.

Do you think that women can bring some fundamental difference in the way we see/manage the product?

I think it depends on the person who gets the job done. What matters is the personality, the field of study and the type of product management and product, but not the gender. For instance, the educational field where you’re coming from can make the difference. Typically, who comes from business management or marketing will have management skills more developed; someone from a technical field will share more about the technical specifications and probably a Product Manager with a background in design will have more to say about user interface. As someone’s told me once, it’s not possible to have all these skills but what is really important is to be able to improve and understand what is necessary to get the job done and to achieve the team’s goal.

Your performance also depends on your company and the working environment. Here, at Premium Minds, I don’t see any differentiation between women and men. Of course, girls are just entering the field and product world is still a man’s world, but this has been evolving and we’re just recruiting more and more women. Recruiting women is happenign naturally, but we should create more awareness to make women understand that they can get into this area.

How can a woman with no experience in product management land the first job in the field?

Software companies like Premium Minds can be an entrance to have these types of jobs. Because the companies that have unique products usually need the product managers to make the roadmap of the product more clear and manage the relationship with the client. You can also approach startups with interesting products that also need product managers. There is also the traditional way to approach the market on LinkedIn, job fairs, recruitment events and companies’ websites. Shortly: be really proactive. Don’t be afraid to apply to the job position that you’ve never done before, even if you think that you don’t have some specific skills.

Lack of information is the main stumbling block on the way to aspire more women to apply to product management jobs

What’re the key skills that employers’ want to see in Product Managers?

Understanding the market needs is the main skill that you need to have, simply because you cannot ship a product if you don’t understand what is the need of your customer. A good PM should also have strong leadership skills. You need to be detail oriented and at the same time have the macro perspective. Try to have the best management instruments to apply into the job.

When you first start a job it’s always difficult to work with the product that you barely know. How to stay focused on your next plans and get the job done?

What matters in Product Management, like in any other job, is managing the priorities and understanding what has to be done now and what can be done later. You should understand what is Minimum Valuable Product and according to that define the priorities.

What part of this job challenges you the most?

It is hard to say, because I like pretty much everything in my job. Struggling against short timings that we usually have is something very difficult but challenging. Another challenge that I have is that I don’t have strong enough technical knowledge. But it is also a plus because it makes me want to grow in that field.

My job helps me to be more organized and care about the details

What do you like best about your job?

I feel a more complete professional and person because of this job. Before I used to have only one perspective on the product. Now I have to think how to find the match point and organize all different fields that can help me put the product into the market. I also like the possibility of applying Scrum and the way it helps my team to work for the same purpose. I am not coding but I am always trying to learn more and more. So I really appreciate the team environment in my company and that I can learn from other colleagues.

How do you balance personal life and professional career?

As you manage your tasks, you’ll also manage your personal life with your career. Of course, the company’s culture and the phase of product development are very important, but there’s always a work-life balance if well-managed. In my daily life I got used to do the same as I do in the professional life. I have some tasks to be completed. My job helps me to be more organized and care about the details. I have to say that organizing and having slots dedicated to specific tasks to be completed is very useful to achieve this goal.

How do you envision a balanced future career for yourself?

I would like to develop my technical skills, that is one of my goals, and also become a better Product Manager through different trainings like Jobs-to-be-Done or design thinking that can help me complete this “scientific’ part of Product management that I didn’t have at the university. I am also trying to absorb the knowledge that is there inside of the company.

Don’t be afraid to apply to the job position that you’ve never done before, even if you think that you don’t have some specific skills

Do you have role models that motivate you grow as a product manager?

Dan Olsen, whose workshop I got to attend last year at Productized Conference, is one of the main role models for me in product management, who helps me understand how to give direction to the quality of the product and how to build the design that can help us stay closer to the user/customer.

There are also some blogs and authors/PM’s that I follow regularly:

http://www.romanpichler.com/blog/
https://www.kennorton.com
http://www.cleverpm.com/
https://www.productplan.com/blog/
http://blog.aha.io/
https://foldingburritos.com/

And of course I attend conferences, masterclasses and trainings. These events help me not only improve my skills, but also share experience with other product managers, network and stay together as a community.

Dan’s Olsen workshop on The Lean Product Process at #productized17

What advice would you give to women who want to become product managers?

Don’t be afraid to embrace something you have never tried before! If you have no/little experience in the field, try to understand how you can make the most out of the skills that you already have. If you really like what you’re doing, it’s doesn’t really mattelr what your background is. Product Management allows you to ‘mix’ the areas and match the lines that sometimes might seem to be parallel. learn

You can learn more about #GirlsWhoProduct here.

I love this image. It’s a simple, yet intelligent, way to explain how we all see the world - with a horn right in the middle of our landscape. This horn represents our beliefs, our concerns and our hopes. Which, as good as they can be, are inevitably blocking our view.

I recently gave a talk at Premium Minds with the title “Giving and receiving feedback”. While waiting for the room to settle down, I had this image^[1] on display. This was my way of reminding everyone that whenever we communicate with each other, there’s a horn in front of our eyes blocking part of our perspective. Bearing that in mind is of the uttermost importance when giving and receiving feedback. This is also true when giving talks.

I started the talk with a fundamental question.

Why give feedback?

If you google 'why give feedback' you will get around 687.000.000 results. That’s a lot of opinions on the theme. Many of the first entries are articles proposing reasons why feedback is so important and how it affects people’s productivity. I’ve read most of them and they have valuable information. I decided not to start with them during my talk. I prefered to go for the common ground that everyone could relate to. So, I asked who in the room enjoyed receiving feedback, even if it was negative feedback^[2]. There was a show of hands. Everyone. Then I asked who in the room gave regular feedback. A few shy hands were raised. Almost no one.

So why give feedback? Because people want it and they are not getting it. Obvious! Maybe not. Why do we want feedback?

The feedback loop

According to Wikipedia feedback as a word was born in the electronics world. It refers to the action of feeding back the signal from output to input.

We have feedback loops in systems all around nature. The body is a great example. I’m only able to write this post while sitting at the freezing Braga public library^[3] because my skin nerve cells send information to the temperature-regulatory control center in my brain, who processes the information and directs my blood vessels to constrict so that heat is conserved. This feedback loop is critical to maintain homeostasis - upon which we depend to survive! I believe, and this is my horn, that giving and receiving feedback is of paramount importance to maintain emotional homeostasis.

The still-face experiment

One day, during an annual evaluation meeting, one of our software engineers shared that he had felt extremely anxious during the first three months of a new project because he didn’t get any feedback about his work. The moment the software architect shared with him his opinion about what he had been doing, even if it wasn’t all positive, had been a great relief.

I think he had the same experience babies go through in the still-face experiment. See for yourself: https://youtu.be/Hh_fW69QkqY

I don’t need to be a baby to experience this level of distress. Just like our software engineer, I just need those that matter to me to not give me any feedback. Of course that, as an adult, I became a professional at hiding my emotions, but underneath the mask, I too start to cry and kick.

You might be asking yourself: Isn’t feedback all about correcting and improving one’s behavior? Why are you talking about emotion homeostasis? Does that even exist?

If you weren’t asking that, sorry for the inception. But I think it’s a valid question. The feedback loop in communication exists to correct and adapt behaviours. However, I think that’s just the superficial layer of feedback; there is something much more important going on at a deeper level.

Project Aristotle - how to build the perfect team

You might have read about Project Aristotle, the Google project to find the answer to the question: “What makes a team effective at Google?”. If you haven’t, start with the New York Times Magazine article. It’s worth it. Of all the key dynamics of effective teams identified by researchers, psychological safety was by far the most relevant to me:

“The Google researchers found that individuals on teams with higher psychological safety are less likely to leave Google, they’re more likely to harness the power of diverse ideas from their teammates, they bring in more revenue, and they’re rated as effective twice as often by executives.” in re:Work

Amy Edmondson, an organizational behavioral scientist from Harvard, defined team psychological safety as “a shared belief held by members of a team that the team is safe for interpersonal risk taking”. You can watch her explain it at TEDx

So, why did I bring up the theme of psychological safety in a talk about feedback? Because my experience tells me they go hand by hand.

When I share with someone what I’m feeling due to their actions, whether it’s great or messy, I’m nurturing the psychological safety between us. I’m taking off the mask and showing there is no secret agenda. And, when I feel safer, it becomes easier to share what I feel. There is a positive feedback loop between psychological safety and feedback.

I believe this feedback loop is more important than the improvement of behaviour. If we foster a culture of feedback, we will feel safer to make mistakes, because we will rapidly know when we have made them. There is nothing worse than learning I’ve done something wrong and no one in the room warned me. It feels really lonely. It also feels that making mistakes is not ok, and as the proverb says: to err is human. Or as Ellen DeGeneres said and I agree:

“When you take risks you learn that there will be times when you succeed and there will be times when you fail, and both are equally important”.

This is the ground where I stand when I talk about feedback. I believe that fostering a culture of feedback improves individual emotional homeostasis which, by itself, enhances team psychological safety, which in turn facilitates the fostering of such culture. That’s the positive loop going on at a deeper level. The exact opposite is also true. No feedback leads to emotional instability which leads to psychological threat, which in turn diminishes the space for feedback.

The way to break the negative loop and start a positive one is always to start giving and asking for feedback. It will hurt at first, but in the long run, it will be healthier.

My talk also had a practical side. I shared these practical aspects of giving and receiving feedback:

Considerations when giving feedback

1. Start by asking if it is a good time to give feedback. This question allows the other to refuse if it’s not a good time. Given a negative answer, you don’t want to give feedback. The receiver will have a huge horn in front of the eyes blocking or morphing everything you have to say. Try again later.

2. Give feedback from an internal space of expecting the best of the other. Remember there is also a horn in your landscape when you observe the other. Respect the other and don’t judge.

3. Focus on behaviours and their consequences. Be specific and take the focus out of judgements. Don’t say things like ‘you are lazy’ or ‘you suck’.

4. It is as important to give positive feedback as negative feedback.

5. Offer positive feedback and… stop there. Don’t use the sandwich strategy. Ever.

6. Don’t build a castle; put up a thousand tents. A beautiful to say don't build up a huge feedback, give it frequently. One of our most senior software engineers started using this strategy of giving frequent feedback. After several honest conversations, he reported to me: “It really felt I was building castles with everyone. It’s so much easier now.”

7. Praise effort, not ability. When you praise effort, you are helping build resilience and determination. Praising ability might result in risk-aversion and heightened sensitivity to setbacks. For more on this topic read the interview with Carol Dweck, professor at Stanford University.

Considerations when receiving feedback

1. Listen

2. Clarify, if necessary

3. Say thank you

Receiving feedback is really simple, but really hard. When someone tells me something that doesn’t correspond to my self-image, it threatens me. I will inevitably shoot out my claws (Wolverine style, but with words) and put up my defenses. My mind will race to find the loopholes in the arguments presented. If I give voice to those thoughts I will start an argument and simply lose the opportunity to see what is beyond my horn. If I just wanted to confirm what I already know, why bother?

Making an effort to just listen and acknowledge is a really powerful exercise. Reflecting on what I’ve just been told will inevitably bring me much more value than the feeling of winning an argument about me. I will also encourage the other to give me more feedback. He will know that it is safe to tell me what he thinks.

Practicing

I finished the talk with an exercise. I asked the audience to form trios, where each element of the trio had the opportunity to give feedback, to receive feedback and to observe the others giving and receiving feedback. The observer then had the opportunity to give feedback to the other two about what she witnessed. I learned this dynamic in a workshop with David L. Patient.

People formed trios with meaningful others. It was a great experience. Someone shared in the end:

“When I realized I was in a trio with two of my best friends I thought this was going to be a waste of time. Now I’m really surprised. I received really valuable feedback that I still have to process. I understood that we are always talking but we never really stop to give meaningful feedback to each other.”

An organization is made of personal relationships. Feedback is a really important aspect of keeping them healthy. There are no shortcuts. ^[4]

Do you have strong reasons for what you’re saying? I would like to start by explaining the main causes that throughout the years developing for iOS led me to this conclusion.

In the beginning (and I’m talking almost Jurassic iOS mobile development back when I first started developing for iPhones), we were working with two OS versions at the same time, migrating from iOS 4.2 to 5.1, the second generation of iPads and the iPhone 4S flagship models were being released. At this point the Drag and Drop kind of UI development available on XCode’s Interface Builder was easy and fast, and very reliable because all the hardware models until then used the same screen ratio and resolution. So the iPhone 4S had retina capabilities, but nonetheless this was just a matter of introducing a new resource/asset size… the “@2x”.

Time passed and more iPhone models and iOS versions were released. Suddenly iPhone 5 brings a strange new large screen and the screen ratios and resolutions start to suffer from fragmentation (this kind of phenomenon was present in Android since the beginning of the OS) and unlike Google’s approach to solve the problem with relative layouts and auto-adjusting view sizes, Apple’s absolute View positioning and sizing was outdated and it simply didn’t work anymore.

So the new models and screens started to create a problem that most likely will continue to escalate as more and more models and screen sizes/ratios come out. (Don’t forget, nowadays we have “@3x” assets/resource sizes and sometimes also the need to introduce some iPhone-6 screen size hacks to make sure the layouts behave correctly on all phone sizes and OS versions)

The release of iOS 6 came with a new way of positioning views that would allow the developers to create apps for all screen sizes without having to create the views by hand, sizing them in code with somewhat complicated formulas to calculate sizes and placement. Apple introduces AutoLayout, a constraint based layout engine and adapts the Interface Builder’s system to support the new layout system as well as the old absolute positioning model.

This is when things started to become shady to say the least. Interface Builder implements the AutoLayout system by creating relations between views and assign constraint values that control position, size, placement. All this is done through “ctrl+click and drag” which is very error prone, because sometimes the action area for the drag and drop is very small and narrow, and is very likely to miss the target and create a relation with some other view, causing the layout to break and the app to crash. Also there’s another small detail, sometimes with just one click and drag, you will be applying more than one constraint to the views. I just couldn’t stop wondering about all this black magic being made behind the scenes.

Another of my main problems while developing for iOS using xib files or Storyboards is version control. The problem here is not that these files can’t be under source control, rather than that, I’m complaining about Xcode as a development tool itself. XCode and Interface Builder are very metadata driven, which means they generate and depend a lot on the metadata created for the project. Each time you touch (even if it is just to see something, read-only purposes) automatically XCode will generate some new metadata and change the file. If you work alone developing the app from top to bottom, this won’t be a problem to you, but if you are a part of a team, this will eventually cause a lot of headaches and a lot of time wasted merging your project’s branches.

So You are saying I should do it all in code?

Well the answer to that question is YES and NO… Let me explain:

You can do all the AutoLayout constraining in code, but if you enjoy your sanity I really don’t recommend it. I do recommend you to get some insight and learn something about NSAutoLayout Constraints and how they work. Maybe do some little experiments with some simple layout needs and get to know them, how and when to use them.

Looking at the image above, we can see that this layout is fairly simple and shouldn’t be very hard to create using code. We only want to get to know how things are done and compare the results.

View placement:
Bottom Green View has:

• width of 90% of the screen width -> widthConstraint3
• fixed height of 100pt -> heightConstraint3
• view center matching the horizontal center of the screen -> horizontalConstraint3
• top matching the vertical center of the screen with 10pt offset -> verticalConstraint3

Top Right Red View has:

• width of 40% of the screen width -> widthConstraint
• fixed height of 100pt -> heightConstraint
• right margin matching bottom green view right margin -> horizontalConstraint
• bottom matching the vertical center of the screen with 10pt offset -> verticalConstraint

Top Left Blue View has:

• width of 40% of the screen width -> widthConstraint2
• fixed height of 100pt -> heightConstraint2
• left margin matching bottom green view left margin -> horizontalConstraint2
• bottom matching the vertical center of the screen with 10pt offset -> verticalConstraint2

NSAutoLayout Constraints

func setupViewLayout() {
    let viewHeights = CGFloat(100.0);
    let viewOffset = CGFloat(10.0);
   
    let bottomView = UIView()
    bottomView.backgroundColor = UIColor.greenColor()
    bottomView.translatesAutoresizingMaskIntoConstraints = false
    view.addSubview(bottomView)
    
    let widthConstraint3 = NSLayoutConstraint(item: bottomView, attribute: NSLayoutAttribute.Width, relatedBy: NSLayoutRelation.Equal, toItem: view, attribute: NSLayoutAttribute.Width, multiplier: 0.9, constant: 0)
    view.addConstraint(widthConstraint3)
    
    let heightConstraint3 = NSLayoutConstraint(item: bottomView, attribute: NSLayoutAttribute.Height, relatedBy: NSLayoutRelation.Equal, toItem: nil, attribute: NSLayoutAttribute.NotAnAttribute, multiplier: 1, constant: viewHeights)
    view.addConstraint(heightConstraint3)
    
    let horizontalConstraint3 = NSLayoutConstraint(item: bottomView, attribute: NSLayoutAttribute.CenterX, relatedBy: NSLayoutRelation.Equal, toItem: view, attribute: NSLayoutAttribute.CenterX, multiplier: 1, constant: 0)
    view.addConstraint(horizontalConstraint3)
    
    let verticalConstraint3 = NSLayoutConstraint(item: bottomView, attribute: NSLayoutAttribute.Top, relatedBy: NSLayoutRelation.Equal, toItem: view, attribute: NSLayoutAttribute.CenterY, multiplier: 1, constant: viewOffset)
    view.addConstraint(verticalConstraint3)
    
    let topRightView = UIView()
    topRightView.backgroundColor = UIColor.redColor()
    topRightView.translatesAutoresizingMaskIntoConstraints = false
    view.addSubview(topRightView)
    
    let horizontalConstraint = NSLayoutConstraint(item: topRightView, attribute: NSLayoutAttribute.RightMargin, relatedBy: NSLayoutRelation.Equal, toItem: bottomView, attribute: NSLayoutAttribute.RightMargin, multiplier: 1, constant: 0)
    view.addConstraint(horizontalConstraint)
    
    let verticalConstraint = NSLayoutConstraint(item: topRightView, attribute: NSLayoutAttribute.Bottom, relatedBy: NSLayoutRelation.Equal, toItem: view, attribute: NSLayoutAttribute.CenterY, multiplier: 1, constant: -viewOffset)
    view.addConstraint(verticalConstraint)
    
    let widthConstraint = NSLayoutConstraint(item: topRightView, attribute: NSLayoutAttribute.Width, relatedBy: NSLayoutRelation.Equal, toItem: view, attribute: NSLayoutAttribute.Width, multiplier: 0.4, constant: 0)
    view.addConstraint(widthConstraint)
    
    let heightConstraint = NSLayoutConstraint(item: topRightView, attribute: NSLayoutAttribute.Height, relatedBy: NSLayoutRelation.Equal, toItem: nil, attribute: NSLayoutAttribute.NotAnAttribute, multiplier: 1, constant: viewHeights)
    view.addConstraint(heightConstraint)
    
    let topLeftView = UIView()
    topLeftView.backgroundColor = UIColor.blueColor()
    topLeftView.translatesAutoresizingMaskIntoConstraints = false
    view.addSubview(topLeftView)
    
    let horizontalConstraint2 = NSLayoutConstraint(item: topLeftView, attribute: NSLayoutAttribute.LeftMargin, relatedBy: NSLayoutRelation.Equal, toItem: bottomView, attribute: NSLayoutAttribute.LeftMargin, multiplier: 1, constant: 0)
    view.addConstraint(horizontalConstraint2)
    
    let verticalConstraint2 = NSLayoutConstraint(item: topLeftView, attribute: NSLayoutAttribute.Bottom, relatedBy: NSLayoutRelation.Equal, toItem: view, attribute: NSLayoutAttribute.CenterY, multiplier: 1, constant: -viewOffset)
    view.addConstraint(verticalConstraint2)
    
    let widthConstraint2 = NSLayoutConstraint(item: topLeftView, attribute: NSLayoutAttribute.Width, relatedBy: NSLayoutRelation.Equal, toItem: view, attribute: NSLayoutAttribute.Width, multiplier: 0.4, constant: 0)
    view.addConstraint(widthConstraint2)
    
    let heightConstraint2 = NSLayoutConstraint(item: topLeftView, attribute: NSLayoutAttribute.Height, relatedBy: NSLayoutRelation.Equal, toItem: nil, attribute: NSLayoutAttribute.NotAnAttribute, multiplier: 1, constant: viewHeights)
    view.addConstraint(heightConstraint2)
  }

As I was saying, it shouldn’t be difficult to get the layout to work, but it is surely painful to look at the code, as well as writing it… (It’s not my brightest code example, but this is only a sample and I tried to keep it as simple as possible)

But wait, I’m not saying you should do all this work, because there are people who already went through the same pain you are experiencing and developed some great tools and libraries to help making it all easier on you. I’ll just refer and give examples of two of them because I have tried it and at some point used them myself. Bear in mind that there are probably many more. You’ll have to do some research and decide which one suits you better, regarding your coding style and layout needs. Here’s a small list of some AutoLayout Domain Specific Language (DSL) libraries that Chris Dzombak compiled to help you get started.

I have used PureLayout and SnapKit and both do the same just fine! Let’s take the example above and see what we need to do to accomplish the same result.

PureLayout

func setupViewLayout() {
    if (!didSetupConstraints) {
      bottomContainer = UIView.newAutoLayoutView();
      self.view.addSubview(bottomContainer)
      
      bottomView = UIView.newAutoLayoutView();
      bottomView.backgroundColor = UIColor.greenColor()
      self.bottomContainer.addSubview(bottomView)
      
      topRightView = UIView.newAutoLayoutView();
      topRightView.backgroundColor = UIColor.redColor()
      self.view.addSubview(topRightView)
      
      topLeftView = UIView.newAutoLayoutView();
      topLeftView.backgroundColor = UIColor.blueColor()
      self.view.addSubview(topLeftView)
    }
    self.view.setNeedsUpdateConstraints(); // bootstrap Auto Layout
  }
  
  override func updateViewConstraints() {
    let viewHeights = CGFloat(100.0);
    let viewOffset = CGFloat(10.0);
    
    if (!didSetupConstraints) {
      //extra view to help setup the layout
      bottomContainer.autoMatchDimension(.Height, toDimension: .Height, ofView: self.view, withMultiplier: 0.5);
      bottomContainer.autoPinEdgeToSuperviewEdge(.Leading);
      bottomContainer.autoPinEdgeToSuperviewEdge(.Trailing);
      bottomContainer.autoPinEdgeToSuperviewEdge(.Bottom);
            
      bottomView.autoSetDimension(.Height, toSize: viewHeights); //heightConstraint3
      bottomView.autoMatchDimension(.Width, toDimension: .Width, ofView: self.bottomContainer, withMultiplier: 0.9); //widthConstraint3
      bottomView.autoAlignAxisToSuperviewMarginAxis(.Vertical); //horizontalConstraint3
      bottomView.autoPinEdgeToSuperviewEdge(.Top, withInset: viewOffset); //verticalConstraint3 - using container view to help
            
      topRightView.autoSetDimension(.Height, toSize: viewHeights); //heightConstraint
      topRightView.autoMatchDimension(.Width, toDimension: .Width, ofView: self.view, withMultiplier: 0.4); //widthConstraint
      topRightView.autoConstrainAttribute(.Right, toAttribute: .Right, ofView: self.bottomView); //horizontalConstraint
      topRightView.autoPinEdge(.Bottom, toEdge: .Top, ofView: self.bottomContainer, withOffset: -viewOffset); //verticalConstraint
            
      topLeftView.autoSetDimension(.Height, toSize: viewHeights); //heightConstraint2
      topLeftView.autoMatchDimension(.Width, toDimension: .Width, ofView: self.view, withMultiplier: 0.4); //widthConstraint2
      topLeftView.autoConstrainAttribute(.Left, toAttribute: .Left, ofView: self.bottomView); //horizontalConstraint2
      topLeftView.autoPinEdge(.Bottom, toEdge: .Top, ofView: self.bottomContainer, withOffset: -viewOffset); //verticalConstraint2
      
      didSetupConstraints = true;
    }
    super.updateViewConstraints()
  }

The PureLayout version doesn’t bring down the number of necessary lines of code very much, but it becomes a little more clear to read and the code seems a little less messy. Now we can see some methods being called that actually express something we can relate to how the view is going to behave in terms of sizing and placement. But we do have a downside with this approach, we have our code separated in two methods. The first will instantiate the views and the latter will do the placing and sizing. As a downside, we did need to add an extra view to help us with positioning the views, but it is only a container to help to achieve view relative positioning. Also we can’t forget to bootstrap the layout engine otherwise nothing will happen.

SnapKit

func setupViewLayout() {
    let viewHeights = CGFloat(100.0);
    let viewOffset = CGFloat(10.0);
    
    self.view.addSubview(self.bottomContainer);
    self.bottomContainer.snp_makeConstraints { make in
      make.height.equalTo(self.view).multipliedBy(0.5)
      make.width.equalTo(self.view)
      make.centerX.equalTo(self.view)
      make.bottom.equalTo(self.view);
    }
    
    bottomView.backgroundColor = UIColor.greenColor()
    self.bottomContainer.addSubview(self.bottomView);
    self.bottomView.snp_makeConstraints { make in
      make.height.equalTo(viewHeights) //heightConstraint3
      make.width.equalTo(self.view).multipliedBy(0.9) //widthConstraint3
      make.centerX.equalTo(self.view) //horizontalConstraint3
      make.top.equalTo(self.bottomContainer).offset(viewOffset)  //verticalConstraint3 - using container view to help
    }
    
    topRightView.backgroundColor = UIColor.redColor()
    self.view.addSubview(self.topRightView)
    self.topRightView.snp_makeConstraints { make in
      make.height.equalTo(viewHeights) //heightConstraint
      make.width.equalTo(self.view).multipliedBy(0.4) //widthConstraint
      make.right.equalTo(self.bottomView.snp_right) //horizontalConstraint
      make.bottom.equalTo(self.bottomContainer.snp_top).inset(-viewOffset) //verticalConstraint
    }
    
    topLeftView.backgroundColor = UIColor.blueColor()
    self.view.addSubview(self.topLeftView)
    self.topLeftView.snp_makeConstraints { make in
      make.height.equalTo(viewHeights) //heightConstraint2
      make.width.equalTo(self.view).multipliedBy(0.4) //widthConstraint2
      make.left.equalTo(self.bottomView.snp_left) //horizontalConstraint2
      make.bottom.equalTo(self.bottomContainer.snp_top).inset(-viewOffset) //verticalConstraint2
    }
    
    self.view.setNeedsUpdateConstraints(); // bootstrap Auto Layout
  }

The SnapKit version of the code is probably my favourite. To setup the constraints we access some of the views properties and assign them values either by relation to other views, or absolute values. In any case, the code syntax is very self explanatory and very easy to read. Unlike the PureLayout, we can have code instantiating the views and positioning/sizing them all in the same place, but we will need to bootstrap the layout engine just the same. One of the advantages over PureLayout is that we won’t need to remember that we have to put the layout code exactly in the “updateViewConstraints” method or in the “layoutSubviews” (in case you are subclassing views with custom behaviour and UI) for things to work.

As you can see from the examples above, you still do all your layout setup through code, but the number of lines written is probably smaller than with the usage of NSAutoLayout Constraints and the readability is much greater using the DSL for Autolayout approach.

As a side note, you can also use something called Visual Format Language to layout your views that allows you to write your layout using an ASCII-art formatted string, witch works fine for some of the UI positioning needs, but you can’t have relative sizing and positioning relations between views which makes it less versatile. You will most certainly have to use it along with NSAutoLayout Constraints. Also it has a special format grammar that you will have to master in order to achieve the same results as using just layout constraints.

TL;DR

Before getting to the point: I do think Interface Builder is a very powerful and helpful development tool. It allows developers to quick and easily start developing beautiful UI and User Experience without even starting to code anything. Also, it allows designers and other non-developers to be involved and able to do some of the UI work without much technical knowledge.

Creating your UI experience through code is not the easiest job, but with enough practice and experience you’ll probably be ending up writing your layouts in code faster than clicking and dragging stuff on Interface Builder and then adjusting the constraints by hand. You’ll have more control over what happens on the device’s screen, manually controlling how views place themselves and adjust to different screen sizes. Also, you'll have to keep in mind all the possible merge conflicts and other multi-member team issues that arise when going through the Interface builder approach.

In the end, it's not all about how you build your layouts that matter, what's important is that you explore and understand how things are done in different ways, and just go with the one that suits your needs. You will be able to make an informed decision and be able to support your reasons for doing so.

Since 2010, I’ve been developing Mobile Apps, for both Android and iOS platforms. For most of these six years I’ve been essentially struggling to find the best approach when starting to each project I’m in.
It’s been a hard and rocky road, each time trying to take my past experiences and turn them into lessons learned on how to make things better.

There are some highly discussed and robust architectural design patterns available for developers to take into account when building applications.
I’ll just refer to the most common (MVC, MVP and MVVM), because these are pretty much the most used and they tend to be coupled to the technologies/frameworks you use and the platforms you target. I don’t want to get into details about them, because that is not the focus of this article, you can get more information on them in Albert Zuurbier’s MVC vs MVP vs MVVM.

MVC, MVP, MVVM

For the purpose of keeping this clean, let’s just take this into account:

MVC – The most commonly used and most basic mobile app development design pattern (android and iOS). It’s actually more M-VC as you tend to have a lot of coupled code that does pretty much everything on the app all in a few layers.
I’m thinking about Android’s old Activities and Async-tasks way of coding, and the iOS Data layers and massive ViewControllers that did all the work of business and user interface logic.

MVP – The commonly used pattern for web applications that some developers tried hard (and with some degree of success) to bring the mobile platforms that work on layering the code in order to achieve some separation of duty and create cleaner code. Though it does offer some more layering abstraction, it still lacks in keeping each layer more focused on a single task. I wasn’t able to decide precisely where to keep the business logic.

MVVM – Microsoft’s Presentation Foundation made this way of structuring code famous, and was brought to the mobile platforms layering the code even more, introducing the ViewModel, which is essentially transfer data, thus removing some dependencies between layers. Yet it presents the same problem as the MVP, lacking focus on separation of concerns and clean data transfer modeling.

A side note to the emerging Functional Reactive Programming, I really don’t know a lot about it yet, but it’s next in my “Study and Fooling around experiments” bucket list.

Searching for inspiration

For my last project started from scratch, I wanted straight upfront to structure things well. I was about to go on with a 6+ months project, and I didn’t want to have my shortcomings of the past to come back and haunt me.
After a lot of digging, I came up to an article about Clean Application development and eventually went about digging even more to learn all I could about this interesting point of view. After reading these two posts I was sold, I wanted to work with this.

My trip for knowledge started with Raymond Law’s article and setup tools for a clean architecture for Swift and led me to the core of the idea from Robert Martin (Uncle Bob). Bob extensively explains in an abstract way his approach to solving the code hell that sometimes haunts us when developing through ad-hoc methodologies, introducing simple layered software as the answer. Each layer is separated using well-defined models for data transfer and interfaces to communicate, while containing and separating functionality.

A new architecture

Eventually and through a lot of idea sharing and discussing with my team mates we came up with a nice clean structure for our code that could be applied for both the Android and iOS versions of the app. This was the resulting abstract architecture.

I omit the worker classes that actually call managers and services and manage all the background/main thread switching. This will be the subject to another article if people find it interesting enough to do so and request for more information about it.

We did stray away from the architecture’s principles by applying the common business layer and data layer as separate classes storing the entire core. We later found out this was not a wise choice, but we were young and wild, and didn’t know the problems we would eventually have by not keeping the modular approach. Having multiple developers working in related screens in parallel potentiated this… Fortunately we practice Code Review and were able to mitigate some of the consequences, but we did spend a lot of time merging code that was duplicated by wrongfully dividing tasks and not going with the modular approach.

We have implemented the apps so that each of the Screens implements the VIP cycle (View, Interactor, Presenter) as a scene, keeping it as logic free as possible. We avoided circular dependencies to keep memory leaks to the minimum (this was specially important to the Android platform), so View, Interactor and Presenter only had forward references to the next step in the cycle. Let’s take a closer look at how we implemented this.

• So the View is actually really dumb. All it does is display information for a specific view model and react to user’s input forwarding it to the Interactor as actions wrapping the data in requests passed as parameters.
• The Interactor is the tough guy in charge of dispatching work to the Processes. Also, he is actually the orchestrator of the work being processed. The Processes keep state and take care of all the logic, but the Interactor has enough knowledge of the business to know which processes to use and how to manage the flow of information and actions between them. Let’s take for example a case where we have to fire calls to webservices in parallel or even use the result from one call in the other as input, the Interactor is the one who manages this kind of workflow.
• The Presenter is kind of a translation layer, where the Interactor sends some information he got from the workflow result and asks the Presenter to display it to the user. For this, the Presenter remodels the information to the essential in order to display in the View., The Presenter then instructs the View to update as a whole (or as AJAX style, updating just small portions of the view).

I know it was a really long post to get to some really small conclusions, but I wanted to give a little background as for why I thought sharing this was important. We have already iterated over this architecture a couple of times (started 2 or 3 smaller projects since then), and although we did make a few changes (as I mentioned before, we brought back the modularity, making the business processes smaller and more contained to fewer functionalities), we are keeping this as our cheat sheet on how to properly structure the apps. Or at least until we come up with a better way… The next best architecture.

Web Components are a W3C standard that is getting really popular nowadays. In a nutshell, Web Components allow us to encapsulate and share reusable components or widgets.
In web development, the combination of libraries and frameworks used in a project is big, as is the amount of testing required to make sure everything works well together. As the application grows over time, it becomes harder to maintain the code and make enhancements. As you can imagine, this makes the development process not as smooth as it could be.

For each project, we create components: a tabbed navigation, a menu, modal windows. None of these are 100% fully reusable. Why, you may ask? Because they have custom CSS styles that may or may not end up messing up the styles in other projects, or the javascript for them to work may use an angular plugin that doesn't work on a backbone project, or they use a jQuery version that conflicts with some other Javascript dependency. So, we always end up rebuilding the wheel. Or a good part of it. And that is really frustrating.

Well, these times are over! For 4 years, since 2011 when they were introduced, Web Components have been carving their path to become the legos we use to build an interface.

What's the point of using these components?

To begin with, you can create your own custom elements which can do just about everything you need. Instead of loading up your sites with too much markup, lots of dependencies and long scripts, you wrap everything up into nice, custom HTML elements.

At this point, you're saying "we've already had this since HTML5 came up, this isn't new!". Well spotted, friend. Today, we use the <video>and <audio> tags all the time, and they are pretty much Web Components. They are usually just hidden from the developer.

So, like the <video> tag, a Web Component is represented by a custom HTML element. This component works like an umbrella, a group of small parts, that enable developers to build reusable components.

These parts are:

• HTML templates <template>
• Shadow DOM
• HTML Imports
• Custom Elements

HTML Templates

HTML Templates are a way of having a portion of DOM that can be reused on the page.
They are a method of declaring reusable markup that is parsed but not rendered until its activation.

The <template> content is suspended until activated beacause its child nodes don't exist in the DOM. It has no side effects (scripts don't run, images don't load, CSS isn't applied) until the template is used.
You can place it anywhere: head, footer, body and since it doesn't load anything of its content, this can be used as a performance hack.

<template id="foo">
   <style>
      /* styles scoped to template only */
   </style>
  <div class="someclass">
     template text here
    <div class="someotherclass">
    <content></content>
    </div>
  </div>
</template>

5 benefits of <template> :

1 - Can have scoped CSS
2 - Invisible
3 - Inactive
4 - Reusable
5 - Importable

Shadow DOM, all about being a ninja!

Shadow DOM refers to the browser's ability to render a subtree of DOM nodes into the document, without including it in the main document DOM tree, while also providing encapsulation for the CSS, JS, and HTML.
So, for example, when you use the <audio> tag:

<audio controls>
    <source src="horse.ogg" type="audio/ogg">
    <source src="horse.mp3" type="audio/mpeg">
    Your browser does not support the audio element.
</audio>

It will generate the play bar and its controls:

But in reality, it has a lot more HTML structure beneath its custom element:

So, as you can tell, Shadow DOM is hidden and not available to the rest of the page, although it can be made visible in Chrome Dev Tools settings:

Only then will it show up as a node in the elements panel:

Entry point to Shadow DOM:

Another part of the Shadow DOM spec is the <content> tag. It acts as the insertion point that projects the text from the shadow host to shadow root.

<!-- Our cool template -->
<template id="my_template">
  <style>...</style>
  <article id="container">
    <content select=".title"></content>
  </article>
</template>

<!-- Our parent doc -->
<div id="host-document">
  <h1 class="title">My title</h1>
  <div>...</div>
</div>

In this example, we are appending the content of the #host-document to the shadow host, through the <content> tag. This is done using the createShadowRoot method.

var host = document.querySelector('#host-document');
var root = host.createShadowRoot();

var template = document.querySelector('#my_template');
var clone = document.importNode(template.content, true);

root.appendChild(clone);

An element that has a shadow root associated with it is called a "shadow host". The shadow root can be treated as an ordinary DOM element, so you can append arbitrary nodes to it.
Source

Since the Shadow DOM encapsulates HTML, its styles wont affect the rest of the page, CSS selectors inside the shadow DOM won't target other elements in the page, and as for javascript, the code running on the page will not be aware of the Shadow DOM either.

So how can we target the Shadow DOM elements and their classes?

* {color: black;}
::shadow p {color: #bada55;}
my-element {
    display: block; }
my-element:unresolved {
    opacity: 0; }
my-element::shadow {
    /*target the shadow root */ }
::shadow p {
    /* any p within any shadow root */ }

For javascript, you can access it via the read-only shadowRoot property.

HTML Imports:

An HTML Import can bundle your JS, HTML and CSS in a single URL. You include it in your page like a normal dependency making it reusable across the whole project.

<link rel="import" href="templateName.html">

The import itself has to be from the same origin, or CORS errors will show up.
The HTML Import can access its own DOM and the DOM of the importing document. The <link> tag must have rel="import declared.
Importing a template doesn't mean you are placing its contents in the document. You're just fetching the file for later use. To actually use its contents, you have to write some JavaScript:

var link = document.querySelector('link[rel="import"]'),
	content = link.import;
    
var el = content.querySelector('.anyClass'); // Grabbing DOM from templateName.html

document.body.appendChild(el.cloneNode(true)); // Appending it to the document

Custom elements:

For a custom element, the big value is semantics. Being able to write <gallery-module> is simpler and more intuitive than filling up the HTML with a bunch of <div>tags that say nothing to those who read them.

How to create one:

1. Name the element: all lower-case and must contain a hyphen/dash. Ex: <fb-feed>
2. Define a prototype
3. Register the Element with document.registerElement()

var FBFeed = document.registerElement('fb-feed');
document.body.appendChild(new FBFeed());

The custom element that you import creates a shadow DOM from a <template>, then registers itself. Imports can execute scripts, so you can use imports to define and register custom elements.

So let's get clear, the goal of Web Components is to reduce complexity by isolating some HTML, CSS and JavaScript code, to perform some function within the context of a page. They let you extend HTML with new capabilities, so you can write better web apps faster.
They also bring value by allowing you to distribute this code cross-browser and cross-framework.

The Bad part is... "So, can we start using them?"

Well... partially. take a look.

The main issue with Web Components is its support. No browser supports them completely. Luckily, there are libraries like Polymer from Google and X-Tag from Mozilla, that act as a polyfill for the missing native browser features. But wait, there's more! No matter if they are built with Polymer or X-Tag, they can work together, and that is exactly the holy grail of standardization.

Polymer is a framework that is based on Web Components technologies. You can make and use Web Components without Polymer.

Conclusion:

In conclusion, we can tell this is a great aproach to standardization, a much cleaner way to create reusable UI elements, with encapsulation and their own logic.
They have been in development for over three years, and all major browsers are actively working on their support, so it's close! Hang in there.
Thanks for taking the time to read this fundamental post and hope you enjoyed it!

Good reads:

W3C Standard

Custom Elements

webcomponents.org

Polymer

React & Web Components

A big Portuguese client challenged us to solve a BAM (business activity monitoring) problem they had.

The challenge

They receive a lot of events and they need to know, in almost real time, if the rate of events (events per second) is outside some threshold. The last 30 minutes of data are the window that is supposed to be considered and there is a threshold for every combination of parameters of events (like an OLAP Cube). This also needs to be checked every second.

The system receives a maximum of 400 events per second, and each event has 8 parameters, giving a total of 240,000 different combinations of parameter values. This is a lot, especially considering that, for every second, they need to check whether each combination falls outside the threshold or not. Moreover, in order to get the rate of each combination, we need to store 30 minutes of data for every combination.

Alternatively, we can store only the events that arrive, and every second, we calculate the rate for each combination. But, this solution is impractical, because we would need to make 240,000 queries every second.

Our hypothetical problem

Let's assume our event is PhoneCall(phoneNumber, countryCode, geoZone). We needed the following combinations:

• all events
• by phoneNumber events
• by countryCode events
• by geoZone events
• by phoneNumber and countryCode events
• by phoneNumber and geoZone events
• by countryCode and geoZone events
• and by phoneNumber,countryCode and geoZone events.

Let's also assume we have 4,000 different phone numbers, 5 country codes and 10 geographic zones. So we will have approximately ~240k combinations ((4000+1)(5+1)(10+1)), and a constant stream of 800 events per second. This is the worst case scenario.

Spark+Elasticsearch for our first approach

Our first approach was to use Spark Streaming. Spark is known for handling huge amounts of real time data with good performance, and it is built in Scala, which I've been liking more and more lately. :)

Spark Streaming works by aggregating small datasets in windows of 0.5 seconds. These RDDs (Resilient Distributed Datasets) are then fed to the Spark engine. After that, you can transform your RDD and a new RDD is returned, so you can pipeline all RDD's through a set of transformations. Check this guide to better understand how this works.

We made a stream which reads messages from a JMS queue and transforms them into PhoneCall events. Then, another stream demultiplexes each event into multiple events according to its parameter values. This will produce 8 events for each JMS message. (Our real case was 60 events for each event.) A subsequent stream uses a [countByWindow](https://spark.apache.org/docs/1.0.0/api/java/org/apache/spark/streaming/dstream/DStream.html#countByWindow(org.apache.spark.streaming.Duration, org.apache.spark.streaming.Duration)) function for each event combination. countByWindow is set up with one 30-minute window every second, and we divide the function's output by 1,800 seconds (30 minutes) to get the average rate for each window. The final stream in the pipeline checks whether that average rate was outside the threshold, in which case, that information is persisted. This is done by using an ElasticSearch cluster. We chose ElasticSearch because it's a fast insert database.

Performance problems

Bottom line, this solution sucks. Not because Spark sucks, but because while Spark can handle enormous amounts of data, it was not made to handle enormous amounts of windows, and this problem requires a lot of them.

Our experiments showed this solution worked well as long as the number of combinations was less than 1000. However, when this number got higher, the lag started, the events started to build up, and the process simply crashed with OOM (Out of memory) :(

Clearly, this wasn't the right solution to this problem.

Let's try a more custom solution

Recently, I enroled in the Principles of Reactive Programing course, and got the idea to use Akka and Reactive Streams to tackle this problem with a custom solution.

Our idea was:

• Have one actor per specific combination of parameter values.
• Have an actor that receives each event, demultiplexes it into multiple events according to its parameter values, and sends each combination to the respective combination actor.
• Each combination actor stores a window of 30 minutes with the number of events with a circular buffer, partitioned in seconds. Each time a second ticks, the buffer rolls one position, and the actor calculates the number of events per second (number of events/1800 seconds), checks the corresponding threshold and sends the data to the ElasticSearch actor.
• The ElasticSearch actor is just an ActorPublisher that sends the data to the ElasticSearch stream driver.

You might ask yourself, isn't an actor for each combination a lot of actors? Well, actually that's something Akka handles pretty well, as you can read from the documentation:

Very lightweight event-driven processes (several million actors per GB of heap memory).

We also did some optimizations, mainly in the combination actors, because this was our big bottleneck when the number of combinations was high.

One of the optimizations we did was, when the 1 second tick is received, we get the last calculated value and we update our total by simply subtracting the value at the end of the buffer and adding the new value. (lastValue - buffer[buffer.length-1] + buffer[0]). After that, we roll the buffer.

Performance results

Well, the results were really nice. With an i7 laptop, we could steadily handle all the events with 800 events per second. Process memory was somewhere around 4GB, which means we could handle more combinations if needed.

Moreover, knowing that Akka can scale horizontally very well, this solution would clearly benefit from this feature. If we need many more combinations or lots more throughput, we can simply add more machines.

Conclusions

This problem was not trivial. We will always need to keep 30 minutes of data for 240k combinations, and each second, check if each combination is in the threshold. It's a memory intensive and CPU intensive problem.

Spark is not as bad as it seems. It has great features, such as horizontal scaling, and it is a much simpler and more intuitive solution. However, it ended up not being a good match to the problem.

Akka is really great for handling CPU intensive problems, the actor model is a really intuitive way of handling concurrency, and when fully understood, it's really easy to shape your solution with this model. Also, because of the way Akka was built, it's a memory lightweight framework.

Recently, I gave a talk with this same title at the Culture@Org Lisbon Meetup, a group of people concerned about their companies’ cultures. This post is a summary of that talk.

skeptic: a person who questions the validity or authenticity of something purporting to be factual. In Dictionary Online

One thing I love about working at a software house is that it's full of skeptics. I believe they help us keep an honest and healthy culture. For me they are a continuous challenge. A good, tough challenge. Here’s why:

Skeptics don't buy bullshit

Skeptics can smell bullshit from a distance. They don't believe in dreams, nor in Hollywood scenarios. They value things for what they are, not for what someone good with words wants them to be. This permanent skepticism help to create a space where all those corporate masks people use to look good simply don't work.

One time we asked the whole company: "Do you know why our customers choose us over our competition?”. You might expect answers like: "We have the expertise, enthusiasm, flexibility and commitment to deliver results time after time within our deadlines." That would be very nice, but that would also sound like bullshit. The reality: 45% of the people surveyed answered that they didn't know. No fancy words.

This honesty makes the weekly questions we pose to the whole company really valuable. With these questions we have a permanent finger on the pulse of our culture’s health. We can spot crisis in the making and act fast.

Skeptics ask the hard questions

When I go to a client I always like to have a skeptic by my side. He will ask the questions that I can't ask either because of politics, or from commercial pressure. He will not only ask those questions to the client, but also ask them to me, which might seem risky. Sometimes it is, but I prefer it like that.

The attitude of really trying to understand what things mean guarantees that we will not sell dreams, and later on deliver nightmares.

This attitude of questioning the validity of things also brings a responsibility for what is being done. A skeptic will not do a task just because I ask them to. He will question what is the purpose and the value of what he’s doing. Because we have a company where we want to grow without managers, this is crucial. Everyone has to be the owner of what they’re doing. That means people really need to understand why they are doing something, and agree that it has to be done.

Skeptics spot unfulfilled promises

When we have a new hire, the very first thing for them is a meeting about our cultural values. We talk about our desired culture: honesty, respect and responsibility. We also ask new hires to be responsible for it.

During this conversation some people raise their eyebrows while leaning back with their head slightly turned. Those are the skeptics. I am sure they will let me know if one day I'm not up to the values I have described to them. That is invaluable for me.

Skeptics will be on the lookout, guaranteeing that the very simple human concepts upon which our culture is based don't just become words on the wall.

In our first office we had several demotivational posters on the walls, they had phrases like:

• Motivation: If a pretty poster and a cute saying are all it takes to motivate you, you probably have a very easy job. The kind robots will be doing soon.

• Worth: Just because you're necessary doesn't mean you're important.

• Achievement: You can do anything you set your mind to when you have vision, determination, and an endless supply of expendable labor. (my favorite)

Some people would read those posters and would not understand the joke. It was our way of saying that we are skeptical of motivational books, speakers and posters. We will not create our culture from posters, we will create it with daily actions. This hasn’t changed.

Skeptics create disruption, which leads to innovation

We have always been very informal at Premium Minds, even on subjects like performance reviews. For some years we have trusted the daily feedback everyone gives and we’d only do one annual review with our Managing Partner. This review was more about feedback than evaluation.

Over the past two years we’ve doubled the team and are now over 75. This model wasn't working with so many people and we decided to redesign it. After some research and brainstorming, I presented a new model during our monthly all-hands meeting.

It was based on a biannual triangular performance evaluation: the product owner evaluated the team and agile coach; the agile coach evaluated the team and product owner; the team evaluated the product owner, agile coach and themselves.

Immediately, skeptics started arguing that this model wasn't aligned with our culture. They said it was too formal and that it created a tight structure that wasn’t always real. They didn’t believed that there should be this specific moment in time where everyone would evaluate each other. It felt fake, they argued. Someone brought the idea of randomness: random people evaluating, random evaluation moments.

When I heard the word random I had a tickling sensation, as if it was a joke. But people were seriously talking about it, and I listen to them. We ended up creating a stochastic feedback model (which deserves a post on its own). This idea of randomness was disruptive and it might lead to innovation within our processes. We're trying it out for the first time this month. I'm very excited to see where it will lead us.

Skeptics are not cynics

Cynics are the evil twins of skeptics. Cynics will question the validity of things not to understand it but to defend themselves. We become cynical when we feel hurt by or angry at something.

A skeptic that isn’t respected by the organization will tend to become cynical. Respecting skepticism is what allows Premium Minds to be constantly experimenting with new processes and tools. We are always questioning if what we are doing and how we are doing it is the best possible way. And we know people will argue not to defend themselves but to defend the truth.

In this constant search we have changed many things and tried many others. Skeptics trust the organization and so are willing to try and iterate to improve things. They are a huge factor for the success of our culture.

The Culture@Org Lisbon Meetup will meet again on Thursday September 24. Join us if you can.

There is a very common feeling among Software Engineers: the feeling of being able to create a tool that can actually cause some impact in our lives, in our society, even in the whole world!

When we start our professional career, we expect to start developing mind-blowing software that could change the way our society works. We also expect our code to be flawless. We expect that the data model and controller we just developed from scratch without taking our fingers off the keyboard will work like a charm on the first try. But then, we are faced with reality: there is still a big road ahead of us and we still have a lot to learn from the people around us before we can actually “change the world”. Basically, our expectations are really far away from actual reality. And that could make us sad. So we go around that and learn how to manage our expectations: we find ways to make sure our expectations are closer to reality. We start considering past experiences and patterns, our own abilities and limitations, and we adapt. How could that work on improving our software solutions and development processes?

The goal

Let's say we are responsible for the development and maintenance of a really important resource management system in our company, which holds many session records in our database created whenever a specific resource is in use. How great would it be if we could use those records to build and integrate into our software a model capable of predicting the availability of a specific resource? That was the challenge! Having already a prototype in its early stages, we embraced the project and started developing together a much more reliable and robust solution in terms of data analysis and processing, and we also made it appealing in the presentation of such data. Then, using all the knowledge gathered, we built a module to do the predictions automatically. And finally, we imported the data and represented the results in a beautiful diagram accessible from our software. Simply said, simply done... or not.

The data

When we start gathering and processing data, we always imagine that the data will be clean and tidy. But, the truth is: if human behavior was involved in the process, it won’t be clean and tidy, and that was the case. We could find all kinds of strange things in our database, such as different users sharing the same resource at the same time. Fortunately, those strange cases are rare and this adds a small amount of uncertainty to our predictions. After spending a lot of time understanding the dubious details of the data we finally got the data clean and tidy!

The key to a predicting model is to understand which features should be analyzed and how. This starts by postulating all variables we want to analyze. Of course we would dream about a very complicated model that can address different variables, understand correlations between seemingly uncorrelated data and predict accordingly. Better to start with a simpler model, and iterate from it. And simplicity says: it depends on time and space.

The analysis

Understanding patterns and trends can be a funny thing to do. You start exploring data, trying your hypothesis, drawing plots and time flows. Sometimes data is nice to you, showing its secrets easily, such as similar patterns that repeat from Monday to Friday, which simplifies our job.

But, patterns are often fuzzy:

In this plot we can see histograms of the number of resource requests for different moments of the day. On the left, there is an ideal situation where the picks are sharp and distinguishable. On the right, there is a real situation where the picks are smoother. That means our prediction is associated with a not-so-small error bar. We have studied statistics, and we know that error bars are to be expected. Nevertheless, it's always complicated to explain this to someone else.

Sometimes, we search for non-existing patterns. We could guess that the resource availability can vary with time, together with some noise. Something like:

On this plot we see resource availability over time. At the top we present three different behaviors, where we can easily spot a trend. At the bottom we present a real plot from our analysis, showing us a noisy reality. The solution is simple, but very difficult for the ego: abandon this postulate.

Anyway, at the end of this analysis we had enough information to rebuild the prototype. And, even if there was some uncertainty, thanks to statistics we could get meaningful predictions. Of course we would prefer to have precise predictions. Just try to explain to your soaking wet friend that even though the weather forecast said it would not rain with 90% probability, the fact of her wetness simply meant she was part of that unlucky 10%.

Presentation

Using the statistical model developed, we could now predict whether some specific resource might or might not be available at a certain time. The goal now would be to find an effective way to represent the data. We have big plans for this feature we are developing as part of our software and as it could eventually be seen and used by a large number of people. It's critical that we present the data in a clear way. The integration of the model should obviously be made without flaws, and that meant we had to start by adapting our solutions on both ends: the model and our software. The reality of building trusty software is not that simple!

However, the topic that was most discussed was the scale in which we would represent the data. We reached a point when we had a very nice diagram representing different types of resources with three different colors according to their availability. It is most natural that our expectations were to have a simple scale from 0 to 100 that could represent the whole spectrum of the statistical model's outcomes. But, again, reality is not that simple and it turned out that during the whole day we had values between 80 and 100 percent for all our resources. We still have some work to do. This project is essentially a team effort and before meeting (or even exceeding the expectations of our targets) which is our main goal, it is very important to try and have control over the “final” result we achieve as a team, which implies that we have to manage each other's expectations.

We still have the goal of combining the occupation rate of related resource types and increase the overall awareness of the user of resource availability across the entire system, thus increasing the utility of the solution developed. Even though we are not entirely satisfied about the results, we can definitely aim for the best we can accomplish with our tools. Our feature is currently available to everyone, which we think to be a very positive thing: we achieved the goal we agreed on. But the question regarding the accuracy of the data represented still remains, and we are still gathering feedback that could validate our results. Of course we will never have an exact representation of reality, but the closer the results are to reality, the more satisfied we, our developers, and mainly our users will be.