top of page
Writer's pictureDaniel Marquez

"The Ghost in the Machine"


Sometimes the seemingly "stupid questions" are the ones that hold the key to solving the most puzzling mysteries. So don't ever be afraid to ask questions, no matter how simple or obvious they appear to be. The sooner you realize that there are no stupid questions, only stupid answers, the more suited you will be to see the problem from a broader perspective, focus on the fundamentals, and begin to think outside the box.

I was brought in less than two months before the scheduled delivery of a complex drilling and production asset to oversee the commissioning of a surface blow out preventer (BOP) system. There are several learnings from this project which you may see in future posts, but there were two seemingly "unsolvable" issues that drive this particular post home.

1. The Possessed Crane

The first issue pertained to one of the gantry cranes, which if you asked those that were involved with it would tell you it was possessed. When you first powered the crane and reset the controllers, the crane would work just fine; if you left it on after use however, the crane would literally begin to move by itself without any commands being issued and ignoring any commands afterwards to try and stop it (except for emergency stop button, or E-Stop). Three separate entities had attempted to figure out what was happening to no avail.

One day while casually discussing encoders during break, somebody asked the current crane representative onboard "Have you confirmed the position reference is correct when the crane is stopped?" As you would expect, the question was met with an expletive-full form of "how dumb do you think I am?!" and stormed out of the room.

Sure enough, the problem that had puzzled many for months was resolved that afternoon. The reason the crane moved by itself was because the position reference changed overtime, so it was doing what it was programmed to do: ignore all commands until it reached its reference position. The cause: the encoder counter was neither reset nor stopped when the crane stopped, causing the counter to overflow, which in turn resulted in the wrong position reference being calculated. This seemingly simple question had been overlooked by many, but once asked, resolved the problem and improved the safety and reliability of the system.

2. The Stubborn Pumps

While commissioning the hydraulic pressure unit (HPU), the pumps were turned on to test run, but when the switch was turned off after the test, the pumps continued to run, ignoring E-Stops. After flipping the breaker in the electrical switchboard to shutoff the pumps, the commissioning team was perplexed. As you would expect, the Client pointed the finger at the original equipment manufacturer (OEM) representative and said (again, in expletive-full form): "What did you do?! This was working fine when we FAT'd [Factory Admission Tested] the HPU back at the shop!"

The OEM representative had not changed anything other than upload a new software version to unrelated controller panels, but being that he was put on the spot he placed blinders on and dove into the code instead of assessing the problem. Trying to be helpful, I asked them "have you checked the wiring in the junction box (J-Box) that houses the switches to make sure it is correct?" They replied that it was checked during FAT and nothing was changed since, so it must be the software and I should let them do their job.

After two days wasted by diving into the software and playing with the pumps without getting closer to a solution, I finally convinced them to open the J-Box. And there it was: none of the control wires had been changed, but the J-Box was opened to connect the electrical power supply. To say that this installation was done violating every electrical code would be an understatement.

The wire gauge was much thicker than required, and instead of changing it or terminating it properly with ferrules, the wires were twisted and jammed into the terminal blocks, snapping the clips holding them to the rail. The strands that did not make it in the assigned slot were left loose, shorting the switches. It was so bad, you could control the pumps by simply pressing the rail with a screwdriver.... Sure enough the broken blocks were replaced, wires properly terminated, and the system worked fine as designed. It may have seemed a stupid question given the circumstances, but nonetheless it helped resolve a show-stopping issue on critical path.

"Is the power plug connected and is the unit turned on?" "Have you tried rebooting your machine?" We see or hear questions like these all the time when we seek support for our consumer products, thinking "do they think I am dumb?" You would be surprised how many issues are solved and how much time is saved by asking the simplest of questions. This is no different at your work site. Lose the fear of asking questions, and you will be amazed how many difficult problems can be solved through simple solutions. Remove the blinders, challenge pre-conceived "certainties", and discover a new angle. There always is.

9 views0 comments

Recent Posts

See All
bottom of page