How a self-trained junior non-developer out delivered an entire IT department.
You read the byline and thought: bullshit, or if you were being generous maybe: hyperbole. Maybe, just maybe, you thought well it might of happened with an exceptional ‘junior’ in age, but ‘senior’ in skills developer. Even if you gave it credence, you figured this was a one off exception that was not repeatable.
Reality is it isn’t bullshit or hyperbole. The junior non developer was not particularly talented technology wise. Further this isnt a one off, this same scenario played out at least four times over the course of a 3 month span.
Even more fundamentally important: this scenario is playing out day in and day out, in organizations both large and small, across all business sectors and geographies.
Forget the model of offshoring to find technical resources at a cut rate price, business departments are moving to a model of just doing the technical work themselves having found enterprise Information Technology resources to be to slow.
Let’s walk through the scenario step by step, and you tell me if this sounds familiar:
Business department X works intimately with popular 3rd party Analytics data. This data is so important that eventually they want to pull down the data from the 3rd party to be able to store it internally and perform more intensive processing/reporting/analytics/visualization of the historical data.
Project Planning They approach their enterprise IT department to do help them out with this. The Business gets directed to the IT Project Management Office (PMO). PMO puts together a project plan, schedules a meeting with the business, collects some semblence of requirements, packages the project plan and requirements together, and puts it in a queue to be prioritized for work by IT management, this takes 1 week. The Business need is not directly tied to a major initiative that IT is being judged by at the C Level, and the Business requestor doesn’t have a personal relationship with anyone in IT to get their need prioritized. So this package of work gets prioritized as low, and sits in the IT backlog for 3 months. Honestly, 3 months is a good deal in this scenario, usually once a low priority item enters the enterprise IT backlog it NEVER gets worked on.
Development Eventually this project ends up on the desk of whatever team is responsible for the enterprise’s Electronic Load and Transform (ETL), lets call them the Data team. The Data team looks at the high level requirements, notices some inconsistencies/gaps in the requirements, and sends it to PMO for a Business Analyst (BA) to work out with the Business. This happens a couple times, taking in total 1 week.
The Data team eventually gets consistent requirments they can understand, and start development of an ETL process leveraging whatever is their standard enterprise toolkit, In this case they are a ‘MicroSoft Shop’ so lets say their stack is:
- Secure FTP for moving the files into the enterprise network
- SQL Server Integration Services to pick up and transform the data
- MicroSoft SQL Server to store the data
- SQL Server Reporting Services to display the report and allow download of the raw data in various formats
The actual development of whatever ETL process takes 1 week
Testing Now its ready to be QA’ed, well not really because of course there is the need to set up Development, QA, Staging and Production instances of the SQL Server database. Including setting up the necessary environments, QA team review of the requirements and development of test cases, and actually executing the test cases takes another 1 week
It passes QA it is done! We just need to move it to Production!
User Testing It goes to User Acceptance Testing at which point the Business gets their first look at their reports, and wouldnt you know it, it just doesnt meet expectations. The report’s don’t lay out the data right, the report takes too long to run, there is duplicate data coming in that needs to be deduplicated, there are tons of missing fields that need to have default values, the reports parameters dont allow slicing the data the way the business needs. Of cource a big sprinkling of plan and simple requirements changes in addition. Got to send it back for some more development, some more QA and then another Business review. Another 1 week.
Change Management Moving it to Production involves getting firewalls opened, whitelisting IPs for SFTP access, shared network drives allocated, domain accounts provisioned, etc. The Data team needs the Infrastructure team to get involved, and the Infrastructure team has its own process requiring change management tickets, approvals from high level management, and the like, so throw on another 1 week.
During the mandated Change Manegement Meeting before the release to Production, the Security hears about the project for the first time. The Security team starts to ask questions. What type of data is being stored? Is there any customer personally identifiable information (PII)? How is the data at rest secured? How is the data in flight protected? All these questions have to be answered and if any gaps encountered remediated. So lets throw on another 1 week.
It finally passes Change Management and is delivered to Production!
RollOut Immediately the process crashes, turns out Production wasn’t identical to Development, and the sample data that the system was designed around wasn’t really representative of data the 3rd party was actually sending. Another final 1 week cleaning all of this up to get it working reliably.
Yeah! Adding all that time up, 5 months from the Business’s perspective between when they engaged IT and they were able to get the process reliably in place.
If you have any experience with Enterprise Information Technology I am sure you were rolling your eyes every single time I mentioned an additional 1 week. You understood this to, first off be a very optimistic estimate for any of these tasks, and secondly a common refrain for late projects. As in ‘Yes, we missed the deadline but we just have to push through 1 more week to get it out. It’s never really 1 week is it?
The project is effectively dead at this point, the business need came and went. The Competitor already pulled in their data, leveraged it, and got whatever competative advantage there was to get.
The Competitor’s Business had already been burned by their IT Department, so they didnt even bother to go to them in the first place. They got one of their younger less expensive knowledge workers to just manually collect the data.
He didnt know any better…
So he just assumed the task was supposed to be simple. He asked the third party to send him emails with the data as attachements. He started getting the data every day as a ‘CSV’ file. His company was also a ‘MicroSoft Shop’ so he imported it into his Office 365 Excel app. He tweaked the data in Excel dropping rows and cleaning it up.
He made some charts but, wanted something he didnt need to send files around with, that he could share as a link in the browser. A couple google searches later and he had his data pumped into Power BI and created his first dashboard. When he showed it to his direct manager, it was so compelling and so cheap that there wasnt a second thought to get approval to get Power BI associated to his Office 365 account.
He was getting tired of constantly uploading CSV files, so he spent some more time googling, found a tutorial and learned how to use Power Automate to upload the attachements of any email he received with a specific subject to Power BI. There was all kinds of weird data issues that were discovered over time, constantly tweaked his Power BI dashboard until it was what his team needed. It took him 2 weeks start to finish. It worked so well he got tasked with 3 other integrations back to back, all of which he was able to bang out in weeks not months.
Eventually the dashboards became widely used and the data became so important that it needed to be centralized, only at that point did they engage their enterprise IT department, who was able to integrate it into the organization’s MicroSoft Dynamics 365 via a few clicks of administrative user interface without any custom development or new infrastructure.
The names have been changed to protect the guilty
The specifics might differ but the story is real, and these scenarios are playing out day by day. Tasks that once required cooridination between experts on multiple enterprise IT teams are now being done by individual technology novices. There isn’t the disconnect and disappointment at the end of the cycle because the business expert is beyond involved in the process he is the actual implementer.
But what about security, what about change management, what about dedicated QA, what about all the patterns and practices of ETL developers who have made this their career bring to the table? All valid points, but fundamentally, these concerns are subservient to the business needs. There is no business leader I am aware of that would prefer to wait 5 months for a dashboard he can have in 2 weeks. In the words of the legend…
If lovin’ you is wrong
I don’t wanna be right
If being right means being without you
I’d rather live a wrong doing life