On Recent Assignments

Lessons learned in contracting and consulting

Advancing Technologies & Professional Goals

After moving to Southern California my roles were in large organizations, as can ben seen in my LinkedIn profile. Some negative experiences and ethical issues within certain organizations caused me to rethink whether that was the correct career track for me. From 2012 onward I made the conscious decision to look for two things in a role: 1) to solve big challenges with modern technology and 2) support and build diverse, high-performing teams. In many cases this meant returning to my original passions - cognitive computing - machine learning, big data and the emerging technologies surrounding that field. All of the roles listed below started with the promise to fulfill some portion of those goals. Below is a reverse chronology on the companies, problem spaces encountered, and notes from each case.


SoundFi (2020)

Like the start-up incubation projects in my early career I had hopes this would grow into an opportunity to build an organization and launch a truly groundbreaking platform. The role brought together my experience with building globally-scaled distributed systems and my extensive background in media and audio. When on partner calls with major studios the CEO referred to me as their "CTO" which was gratigying to hear. But it was still a small company with a "killer app" as its core product. The downside? Although it was a promising start-up, there already was a considerable amount of technical debt from previous software vendors. Then there was the Herculean task of pivoting from in-theater delivery to streaming online.

While the shift was successful by many measures, the COVID-19 pandemic along with some business decisions that pre-dated my arrival limited the options for continuing. I wish the best for SoundFi and am grateful that Chris Anastas has provided a strong recommendation for my work there.


Insight Consulting (2019-2020)

I was brought in to this role in order to anchor a new office for Insight's "Digital Innovation" division in Los Angeles. The cornerstone project was a multi-year predictive deployment project with a Los Angeles County agency. One phase of the project was to provide near-real-time data based on in-coming emergency calls. The other leg of the project was to do deep historical analysis to help determine optimal placement for new permanent station facilities within the county. So the project had both short and long-term agendas - interesting, strategic stuff. There were also projects in the works in healthcare and IoT, but the initial focus was on this government project.

This role had all of the hallmarks of what I was looking for - the promise to build a diverse, high-performing team on a project of real community benefit. What actually transpired was significantly different. After I joined Insight I learned that the agency had not officially approved the project and I was redirected to work on a data warehousing project for a multi-national paper company based in St. Louis. I took it as an exercise in "going with the flow" and biding time until the cogs of the government wheels were in place. But it allowed me to see how Insight managed client companies. Unfortunately the paper company assignment was laden with perverse incentives and toxic consulting behavior equal to any project I had witnessed from either the "buy" or "sell" side of the vendor relationship. This was not what I had signed up for.

I expressed my misgivings to Insight management - both about the St. Louis assignment, the status of the LA-based project and how Insight ran projects in general. The senior director for the department traveled to LA to meet with the county agency and afterward had a sit-down meeting with me. Basically it boiled down to "this is how consulting works", albeit over a long meal with expensive drinks and fine food. I tendered my resignation that Friday.


Kaiser Permanente (2018-2019)

This was a consulting role with a hire option that had many of the hallmarks of my time with TherapyEdge - opportunity to make a difference by bringing new technology to the practice of medicine. Kaiser Permanente was in the midst of a $300 million overhaul of their IT infrastructure to modernize systems and processes. While this role worked with non-clinical data, the importance of what they were studying - hospital utilization - had wide-ranging implications for delivery of care for millions of people across the US. And of course as a major nationwide healthcare provider those changes usually have a positive ripple effect to healthcare delivery in smaller markets.

What I didn't know until I arrived was that I was hired into a "shadow IT" group that was trying to push their part of the project forward despite a middling effort by KP's core IT services. This is never a recipe for success, but I made a good faith effort to work with the challenges in front of me. I later joined a master data working group with the Chief Data Office - only to find out that there were regional and IT executives that were arguing against the very existence of a Chief Data Officer. As usual, this situation is like so many others in my 20+ year history, the IT problem was really just a symptom of a larger institutional problem.

Shortly afterward our working group's executive sponsor from the CTO departed KP, and I realized then I wouldn't be far behind. I was offered full-time positions within several divisions but didn't see any of those situations as a good fit. During my time there I was able to hire in contributors to a variety of teams that are still at the company, and I made many long-term friendships there. Several of the people I hired into KP have provided recommendations both on this site and on LinkedIn, and I'm optimistic that with their skills and temperament they'll continue to excel.


Attunix/Redapt (2017-2018)

This role provided a chance to do a deep dive into Azure's machine learning infrastructure. Attunix had been recently awarded "Partner of the Year" by Microsoft and the company was looking to expand their advanced analytics practice in the Southwest US. I was finishing up the capstone project as a Microsoft Certified Professional in their data science track and it seemed like a good fit.

I was initially assigned to a government project and was a contributor to Microsoft's Data Platform Advisory Group. I was doing some "dog food" projects in Azure in conjunction with their engineering teams. This included working with R.NET in Azure Function Apps, and I was seeing a lot of potential there over HDI and other "under the hood" technologies they had in the works. I also got to build and deploy and a layered fraud detection engine for a national medical billing company and designed an IoT solution for the energy sector. Beyond that I was also presenting on machine learning and hybrid infrastructure patterns at Ignite conferences on behalf of Microsoft - and life, as they say, was good.

Attunix was then acquired by Redapt, a company known up to that point for building private and public cloud infrastructure for other companies. With changes in the org due to the "merger", along with my growing connections within Microsoft circles, Redapt assigned me through Microsoft Consulting Services to a large Databricks project for T-Mobile. They were preparing for the Sprint merger, and that team I worked with was tasked with processing data from seventeen different phone networks to standardize their customer service data.

While that was going well on a technical level, the organization at T-Mobile was like any other big conglomerate where I had worked - too many competing factions that were interfering with the actual heads-down work. Between Attunix/Redapt office politics and six months of wading through the low-grade toxicity at T-Mobile - amplified by the impending Sprint merger - I took my leave. When the SOW ended with T-Mobile I left Redapt.


Advantage Solutions (2016-2017)

This was a contract-to-hire position with a national marketing conglomerate, and I fully understood that there were risks going in. They had the classic problem of ingesting huge daily volumes of data from Nielsen and other providers. The largest system there was wedging feeds into a data warehouse riddled with 5th normal form data structures and it was buckling under the weight of the vertical scale. In the interview process I advocated for a semi-structured approach, providing Hadoop cluster and data lake implementation options. The VP of the group was very enthusiastic - but oddly the interviewers from that working group were actively resistent to the idea. On balance I took it as both a technical and team management challenge.

I learned soon that ASM was the product of several mergers where the technology orgs from those disparate companies had never been integrated into a unified organization. That meant each silo had been passively conditioned to undermine the other, as though they continued to compete with each other as separate companies. They only collaborated when it came time to oust a fresh voice from outside their carefully curated boundaries. This time it was me.

Eventually I was sidelined to an internal project - ironically - to look at seasonal and temporary employee turnover. Through analysis of the data - and interviewing field managers about their practice - it was found that it wasn't the quality of the applicant, but rather a combination of geography and how they were assigned work that determined how long they remained. I presented the findings on the story the data was telling and analysis of the field interviews - also backed by the data. The final artifact was a remediation plan for limiting the managers "gaming the system", and suggested additional points of data to be collected in order to better track behaviors and outcomes. Executives shelved the project and I was let go. And as a coda to this episode, the VP that hired me left the company a few months afterward. So the team management challenge was at a level in the organization neither I nor my boss could address. Lesson learned.


Mattel, Inc. (2015-2016)

This was an assignment with a firmly dated SOW. I was selected to manage a group of KPMG contractors due to my financial management background and knowledge of Hyperion. My cloud experience also caught their attention, as they were looking for ways to streamline much of their reporting through PowerBI, and I had spent some time with Azure while at Universal Music Group. This role with Mattel had both process compliance and data management facets to the role, so it had a balance of the tried-and-true with a bit of the new-and-shiny.

The interesting part of this story comes from the backstory. Previous to this project the CEO and CFO had been removed by the board because of accounting issues that caused a potential acquisition/merger with Hasbro to be nixed. To add insult to injury, their servers housing the ledger data were so out of date that Microsoft refused to continue supporting them with security patches. Mattel was literally the very last Windows Server 2000 customer on planet Earth. This is all known in public disclosures following the changes on the executive level, so I'm not revealing any "inside politics" here. That project was a direct response to those events, and resulted in a full re-architecture of their internal ledger system.

I wrote an anonymized treatment of a portion of the problem space, focusing on alleviating consolidation bottlenecks with ledger data submitted from an array of global manufacturing locations. Although they were in a wide variety of time zones, their practice was to submit ledger data at the last possible hour (Pacific Standard Time) of the given reporting day. It's a time capsule both from the perspective of early cloud adoption by large companies and in how to orchestrate a global financial reconciliation process. They kept many manual processes, again, as an institutional reflex - but the team eventually delivered a relatively modern, GAAP compliant system.


Honda R&D US (2014)

I was really hopeful for this project, as it combined my experience with mobile application development, IoT and cloud-based systems. Honda had a telematics system that was purpose-built for limited scale of data traffic - designed for their electric and hybrid vehicles along with the full Acura line. I learned a tremendous amount about electric vehicles, and the experience reminded me of my time working for Bob Moog - but on a much bigger, more industrial scale. I spent as much time in the R&D "bench area" as in the office, so I felt at home in that environment.

The problem space included two general factors: 1) the current infrastructure was buckling under the current load, and 2) Honda was planning to include all vehicle lines on their telematics backplane in future model years. The first part of the exercise was to calculate what the "data traffic" would be over the 3G backbone, and estimate the data payload and volume for the lines that would roll onto the system.

After building a simulator that showed how the system would not perform under current bandwidth constraints, I then made a recommendation for a cloud-based queueing system which would also land data in a NoSQL store. One of the dirty little secrets of the on-prem solution was that the structured data store was being truncated by the IT team. This was data that needed to be kept for 7 years to comply with NTSB regulations, and the data store preserved less than a year's worth of records. I later learned that the executives over the group rejected the idea of expanding the hard drive space because the system vendor wanted to charge Honda $5000 to swap out a hard drive in the data center. In the context of a potential adverse event investigation that is a pittance - but they chose not to see the situation that way.

By request I outlined a new platform that put the entire stack in AWS, using a cascade of services to handle queueing during high-usage periods, and would have enough storage capacity to both satisfy NTSB mandates and provide reporting both for compliance purposes and for showing performance of various vehicle lines. I was fired a week later.

Afterward I learned that the executive in charge of the group was a former contractor that previously ran the IT team currently truncating the data stores. There were similar "sweetheart" connections to the company managing the "private cloud" contract. A year or so later the entire working group was disbanded and a different vendor was selected to build a new solution.

Their recommendation? A NoSQL solution built in AWS.

I have since created my own NoSQL telematics proof-of-concept in Azure, and have a blog entry here that's still in the editing phase. That article will be published soon.