This is a two-part blog post (or maybe the format is more of an article…) about what the equivalence of code tests and take-home assignments can look like when you start interviewing for manager roles. In the last half we looked at case studies and dilemmas. When I published that, I got some excellent questions that I’d like to start off with addressing. “Would this same approach to manager interviews also apply to leadership interviews”. My short answer is: Technically you can use this type of case/exercise for any process, just adapting the case or the questions and how you review it to match something relevant to the role. That direction is easier, I think. Designing a practical hands-on exercise in the style of say a leetcode test that targets manager-skills I believe is harder. I do assume that this is what the email exercises I mentioned in the other part was meant to do. (I am so curious about that one! I almost wish I had gotten to do one of those)
In this second half I want to look a bit more into another type of “challenge” I’ve had to do: the assessment questionnaires, or maybe you could call it pre-interview questions.
Candidate Assessment Questionnaires
This is basically a list, long or short, of tough (mostly) questions that you reflect on and send in answers beforehand. Preferably without ending up writing a book.
One could see them as one step up from the questions companies have started to put in the application form to use as a first filtering, taking over from the cover letter which seems to be dying out. These filtering questions have to be pretty few and “simple” and the interviews are usually too short to be able to get very deep into someone’s experience and personality and assessment questionnaires are one way to try to get around that. They give the employer a chance to get more information out of the candidate and leave room in the interview to go deep into some areas, maybe dig into some interesting reply, or leave entire parts if it is obvious the candidate has great (or no) experience in it. The aim of the process is to figure out if this is a person that can do the job but also that they will be someone you (manager, team, organisation) will enjoy working with.
For me as a candidate, I actually really like these – even though I struggle with them. They give me the opportunity to reflect, articulate why this is something I will excel at (or not) and it gives me a lot of information about the company, the role, big challenges up ahead but also things like:
- Do they share my beliefs? If not – Is it a belief important enough that I will be miserable having to work against it?
- Will I learn and have a chance to grow here?
- Do they understand what (if anything) needs to change to get the result they want? Otherwise I will not have support and will fight an uphill battle.
But as I mentioned, I also struggle with them, for a few reasons.
- I am not used to thinking in the terms typically used. I am someone who would be called a “doer”, I have to consciously force myself to not go straight to action mode. That is excellent when you are an IC and works ok when you are a first line manager but it does not really work when you are looking to move into higher manager-roles.
- I am not a person who writes short answers. I tend to really like my own voice, and love my own writing. But I also know very well that someone needs to be able to actually get through the replies without falling asleep or running out of time. Just as I would not If a candidate sent in essays for something intended to be answered in a paragraph, I would not be happy. (same goes for during the interview by the way. Rambling for too long usually is not a great sign) Being able to go reasonably deep but stick to the topic and be concise is a great skill to hone.
As with the case studies – I try to find hints and information in how the question is phrased and the combination of questions, find something I’ve previously worked with, talked or written about or have recently read about that I can use as an anchor. I make sure to tie back to previous questions and answers when possible and take notes for the potential interview about any reflections or worries I have. One process had a lot of questions about challenges between team managers and product owners – to the point I felt there must be a huge issue with unclear roles and responsibilities. I made sure to bring that up in the interview. Another very clearly showed that there was an issue with modernising testing and a “us versus them”-tension between developers and testers. This made for some excellent discussions during the interview so I strongly encourage you to reflect on the chosen questions and bring up points that made you react, either in a good or bad way. If you can – use skills and personality traits mentioned in the job ad to tie everything together and show you’ve understood what they are looking for. And also – if some of the questions are not your strong suit – that is ok to say! You come across as honest and trustworthy and if you can also add some thoughts and ideas of how you could learn, or how you would go about figuring it out.
The assessments ranged from maybe 6-7 questions up to 15. They have ranged from more light, questions that could have been part of the actual application process, to really deep and heavy ones. The work I had to put in ranged from about one hour up to almost a full work-day (including a few reviews and re-work). This effort is not unlike what I’ve put into the cases and other home assignments I’ve had to do. Which of course raises the question – how much work and time is acceptable to ask of a candidate? The balance between getting the match as good as possible and not asking candidates to put in a lot of free work is hard. I know from the later interview that my work on the biggest assessment was more than what they expected – but from my point of view I kept my answers as concise as I could, they just asked (a lot of) very big and difficult questions.
The questions can be broadly grouped into a few areas:
- You: Who are you and why do you feel excited about the role and/or company. Future ambitions.
- Strategy work: Questions related to long-, mid- and short term planning, how you align your area of responsibility and its needs and wants with business goals. Can include things like scaling an organisation, risk management.
- Delivery: Your experience working with things like roadmaps and driving initiatives (technical and/or possibly product initiatives depending on how the role is defined). How do you balance technical initiatives and work with customer facing and/or business related work?
- People/team management: These are questions related to figuring out who you are as a manager. What’s your leadership style? How do you build healthy teams? How do you build a good culture?
- Technical Leadership: Things like how to set/drive technical vision and strategy, working with tools and adapting new technologies. Being data driven and using metrics, KPIs or similar.
- Collaboration and communication: How do you work with different stakeholders? How do you work with vendors and service providers?
- Resource and budget work: Questions around financial planning and follow up. Resource planning and allocation.
With every process I try to put in as much effort as I feel is warranted. How badly do I want the role? Can I learn from the work they ask me to do? I would not, when on the other side of the table, ask someone to put in the amount of work I did in the biggest of these assessments, I do feel like it was too much. But at the same time I feel like I learned a lot from working through these – so I chose to do it. It also did make the following interview immensely more interesting and useful so I do see the gain from it. I think in the end I prefer this approach to a process with a lot of interviews with different people where you spend just as much time in the end, but never manage to go deep. It’s the Goldilock Heuristic again, isn’t it? We don’t want too much, not too little – we want just enough.
But let’s look at some examples of questions (anonymized and slightly rewritten as per request from the employers in question) and how I chose to answer them at that point in time. I’ve tried to pick different types of questions to show how much they can differ. If you are not interested in the example questions and answers – you can scroll down or go directly to the takeaways here.
Example questions
What are your strengths at work?
An ability to “connect the dots” – identifying gaps and risks and how it connects to the bigger picture.
A ”doer” – someone who finds ways to drive things forward and unafraid of making decisions
Genuinely caring about people and wanting them to be their best. Everyone shines in the right place and with the right support.
I am not afraid to bring up uncomfortable things and point out problems.
I think we need leadership that will educate and champion modern agile development in the rest of the organisation. Empathy and genuine care for everyone in our department.
Additionally, a lot of guts to stand up for what we believe in, daring to take difficult conversations and communicate, communicate more and then communicate everything again – so we don’t leave people behind.
What caught your interest in the position and what do you think you can bring to the role?
I have had the opportunity to see the work X put in, where they focus and the strengths and individual flair they bring to it and I think this would be an incredibly rewarding and fun challenge to take on. The organisation is in a healthy place but with a lot of interesting challenges to tackle and I would love to drive that work and grow in that direction. I bring new perspectives from working on several different “crafts” and businesses during my career. I have a very deep technical knowledge from a lot of different systems and organisations, have been part of everything from a small 20-ppl company up to working on modernising a very old and traditional business. I have experience working with steering groups, driving large projects and even setting up new business areas. I am comfortable with working on both software, developer experience/platform and infrastructure projects. I have excellent skills when it comes to working with risks and improvements and I believe I have already shown that I have a drive and passion for how to build great software.I have already shared what I see as our biggest challenges getting to the next level and I would love to get to actually drive those questions forward. I think I would balance the management group in a very good way.
I would like to work more with developing leaders – it’s something I miss doing. Working with other managers (or other senior leaders with similar problems) have been the most rewarding manager-employee relationships I’ve had in the past. I miss working on a department level with more tactical/strategic focus – something I focused a lot on at previous workplaces.
Have you been part of any digital transformation initiative? How would you build a strategy to integrate quality practices into the overall company strategy?
To simplify: I would be looking to get testing to shift left as much as possible and make testing a team responsibility while also shifting right as much as possible – making teams see running their services as just as important as building them. Depending on the current state and main pain points my prioritisation and initiatives would differ a lot.
In a company where the risk appetite was very low and we were still very early on both our agile journey and our journey towards modern product development – I chose to focus a lot on shifting left. This meant a lot of focus on integrating testers and requirements specialists into teams, improving automation, removing silos and working on educating people in the organisation on agile ways of working and what good testing looks like.
It is a continuous journey of improvement that I am sure they are still on, but I would say that by the end of my time at the company – we had shifted testing into a natural part of the team, testers were seen as valuable assets to include all the way from initial discussions into actually releasing features and my favourite part was having shifted the view of security testing from something expert did at the end (which I might add is still valuable, I just believe we can let the experts focus on finding tricky problems, not basic things we can check ourselves) to an integral part of software development (checking for basic OWASP problems, doing static analysis of code and knowing how to test for things like SQL injections)
At another company I got the opportunity to see that in a company with a very different level of risk acceptance – with the right setup, mindset and people, “testing in production” can work splendidly. If you can identify problems before users notice them, and rectify them in minutes – you don’t have to cover all bases before releasing. It requires a lot of the team, and the business, but in the right situation it is most definitely the right choice. Having proper monitoring, alerts, teams taking ownership and “thinking DevOps” is the foundation to that. And from a SRE-perspective – having proper SLO (service level objectives such as uptime and response time) and SLAs (service level agreements – the promise we make to others about the SLOs) so people know what they can expect from us. In this scenario we focused on building up testing competence in the team but invested a lot in SRE-related skills.
Do you have any experience building high-performing teams?
I have had roles that span a wide range of types of leadership, with more or less direct and indirect influence of team structure, processes, ways of working and actual work. Depending on the role, the type of work I could do to inspire this differed a lot. From acting as an informal leader in a team and focusing on improving ways of working, learning from mistakes (and successes) and nudging people in a direction I believe in. In a more traditional line manager role, with little opportunity to work and observe within a team, I’ve had less opportunities to influence the team directly but ample opportunity to work through others and influence the bigger picture, plus it gave me access to being able to shape the setup of the teams. And in the role I have right now, where I have two teams where I am deeply involved in everything from technical decisions to what we prioritise to who we hire – it’s mostly a matter of making sure I lift up others and show ways of doing things better.
Or to summarise: I believe the base to high-performing teams lies in
- A clear vision, mission – the why – that the team actually cares about
- A good diverse set of people
- A good level of autonomy with clear frames to operate within
- A healthy workload and reasonable cognitive load
- An environment with high psychological safety where it’s safe to fail and encouraged to experiment as long as we learn
- A sense of ownership and accountability – we take pride in our work and regardless of who was responsible for, or executed, a task – the team feels accountable for the result.
How do you foster a culture of innovation, collaboration and continuous improvement?
In a lot of ways, the same way as how to build high-performing teams.
- Feedback: Often, actionable, with the intent of helping each other
- Psychological safety: If you’re safe to be your authentic self – you are likely to show more courageous behaviour
- Encouraging collaborative ways of working, like pairing (or more), shifting left/right and viewing quality as a team responsibility rather than the responsibility of a tester (this of course goes for other things than testing – I am simplifying here)
- Retrospectives, both regularly but also after bigger wins and resolved issues. Again: It is ok to fail, but what did we learn?
- A way of working that is based on experimentation: Here lies things like breaking things down into small enough parts that we can try them out without feeling like we can’t change our direction, making space for trying new things, making sure we try (several) things and don’t assume we know the right answer (as an example – use techniques like A/B-testing, canary releases, chaos engineering).
I guess this could be summarised as a focus on experimentation and continuous learning: It is ok to fail and take risks as long as we always focus on what we learn, and use that to change and adapt.
How would you go about evaluating and adopting new tools and technologies?
Tools and technologies are great but in the end – it all boils down to the problem. What is the problem we are trying to solve? Once we have that, you can use a lot of “standard” questions to build up your list of requirements. Things like: What are the must haves, the should haves, the nice to haves? What legal and regulatory limitations are there? What is our budget? What technological, architectural and organisational frames do we have to take into consideration?
From that, you can use any of a multitude of processes or frameworks to gather data about the available options and make a business case for one or two top candidates. Identify (needs) – Explore and evaluate (what options exist and how they meet the need) – Decide (moving into the procurement process)
My main point here is probably that it’s important to be aware that there usually is no tool or technique that is best in class in all areas, meets the budget and has everything we want (which is one reason we have so many in-house built tools solving standard business problems) so sticking to the main problem is key.
How would you prioritise resource allocation to make sure you get the biggest impact while staying within budget constraints.
Tying back to a previous question – there will always be more work than available man hours/money. The long-term strategy should, in my opinion, focus on whole team responsibility and enablement – coaching, teaching and building tools and frameworks. Of course, there will be tasks that are better done by experts – but that should at least be the mindset in my opinion.
Depending on the current state – that could be the short-term strategy as well, or that would have to be a multi-year process to move in that direction. If the current state is that a lot of testing tasks or SRE implementation tasks needs to (or should due to circumstances I am not aware of) fall on people in the QA and/or SRE-roles – I would look at a short-, mid- and long term plan in order to avoid as much reactive work as possible. Having to make acute decisions on priorities is usually not where the best decisions are made. I would need to know more about the current state to be able to know which factors to optimise for but potential techniques to figure it out could be things like risk analyses (where does it hurt the most?), impact mapping (where would our efforts move the needle the most?) or just looking at urgency (are there projects where there is a set deadline that we cannot avoid working on).
My end result would likely strive to make sure we have clear goals, use a “coach first” approach, teams feeling and taking ownership over their own quality and stability metrics and efforts, less issues with less individual and overall impact and faster value to end customers.
Takeaways
When I first faced one of these, it felt almost impossible and I had to spend so many hours answering. I struggled understanding if they wanted short answers or very long ones. I struggled finding relevant examples, if they were not a 1:1 match. I felt defeated and like I could never be the right person for the job.
I am incredibly grateful for my network and friends who have helped me think, reflect, sometimes just whine, and then work on translating experience into valuable stories. Ashley Hunsberger helped me work through these kinds of tasks more than once and has been such an important mentor for me in this aspect of management. Thank you so much Ashley! I’d also like to send a big thank you to Isabel Evans for helping he think through this particular post and making me think of one to three additional posts on the same theme for the future.
I learned a lot from looking at how companies chose the questions and how I could use that to my advantage in the interviews. I would have loved to have something similar to this early on and I hope someone out there will find this useful. If not – I have, as usual, learned a lot from writing it and I must say I really like a lot of my answers! I would for sure have hired me.
If you disagree with my replies? That is totally fine. Depending on day, my current focus, how I am feeling that day or just As I wrote in the introduction to one of the assessments: “A lot of these questions are difficult to answer without having more context, because the answer will largely depend on things I don’t have the knowledge about yet. Things like which goals are most important right now and which will be important in the next 2-3 years. Which metrics to optimise for, and how much it’s ok for other metrics to take second place. In the end – building software is constantly a balancing act between different perspectives where we cannot get everything we want. Basically, we will get what we measure, and optimise for.”
Good luck out there and I hope you nail those processes my friends.
