Automated Bathroom Attendant

Introduction

Flushing the toilet and washing your hands are two things that always go together. We're basically trained at birth to make sure you flush and then you make sure you wash your hands. For 99% of the time, this probably isn't much of an issue but for that 1% of the other time, it may be. That 1% of the time probably includes a distraction such as a phone in your hand and your brain doesn't always tell you that you successfully did that thing that you always do. You could wake up half asleep and go to the bathroom, lay in bed and then think, "did I flush the toilet?" and then get back up to check just for you to be like, "well yeah, of course I did." This fun game that has never happened to me personally but I'm sure is something that has happened to others. This could be solved with a nice over-engineered solution. It'll rely on the hottest technology currently out there - cryptocurrency. Just kidding, it's AI!

The Plan

It's pretty straightforward I think. We take our Infineon’s PSOC™ 6 Artificial Intelligence Evaluation Kit (CY8CKIT-062S2-AI) and record a bunch of toilet noises and sink noises and then identify them as such. We use the accelerometer to detect whether or not the door is closed (maybe we'll use a door sensor but I feel like an accelerometer is good enough for now).

I'll break down the categories of what we are looking for:

FlushHeard - we are going to use the microphone to detect whether or not the toilet was flushed. I'm not going to get into whether or not we can detect what the person is doing on the toilet because THAT'S INVASIVE! Whereas listening to a flush sound is like whatever.

SinkHeard - we're going to use the microphone to detect whether or not the sink was turned on. I don't feel like we need to tell someone that they didn't wash their hands for 85 minutes or whatever because that seems kind of nanny-state-ey, and I'm not that into it. I just want to know if I remembered to wash my hands.

DoorClosed - We're going to use the built in accelerometer to see if the door is in the process of closing (or opening). This will be helpful to determine when we want to send an alert.

AlertSent - Idk what kind of alert I'm going to use but if I can get IFTTT or equivalent working, I'll do that. Otherwise, just logging that will be helpful.

TypeOfAlert - We have four situations - 1) Nothing to see here folks (the default), 2) Did you go into the bathroom, spin around and leave?, 3) Did you forget to flush (get back up and go flush) and 4) Did you forget to wash your hands?

Time - If time allows, I'm going to factor in the time for this device. I really only want to know scenario 3 - did I forget to flush, at all times of the day. For the other ones, it really only comes into play between the hours of 11PM and 6AM because that's when my brain is asleep.

We're going to use the device and plug it into a battery and if time allows, make a little case for it to sit into. I'm going to stick that onto my bathroom door and then we'll be good to go!

Placing the Device in the BEST SPOT

I want to eventually place this device on the inside of the door so I'm recording all the sounds from the inside of the door. I'm going to use the best adhesive known to man - sticky tack (I try to always use the name brand Loctite which I am disappointingly not sponsored by [could you imagine Loctite Blue Sticky Tack thinking they need to advertise?]).

Getting that Sweet Sweet Data

So the most important part of this project is theoretically the hardest - making a machine learning algorithm to figure out if the sound the device heard is a flush or a sink noise and to ignore all the other noises around.

Using DEEPCRAFT (TM) Studio and plugging in the PSOC™ 6 Artificial Intelligence Evaluation Kit (CY8CKIT-062S2-AI), we can create a new project and call it something like random like Bathroom Recorder and select the template (Templates>Graph UX>Generic) and click OK!

This will create a random blank window which we'll drag in the Microphone under Boards>PSOC™ 6 Artificial Intelligence Evaluation Kit (CY8CKIT-062S2-AI)>Microphone. We'll also drag in the Data Track under Library>Visualization>Data Track and then connect the red little vector thing into the input. We'll also drag a Predefined Labels from Library>Meta>Predefined Labels and then change it so it says Flush and Sink.

From here we can press that play button (it looks like |> but you know, with the arrow filled in) and that'll open the session window. We can press the record (looks like an O but you know, with it filled in) and then it'll start recording. There's a little Flush and Sink label next to Labels: that you can click on as you record. This will help you label as you go which eliminates having to listen to the sound of the sink twice (although you'll want to adjust the labels as you go so will need to listen twice. Maybe not thrice!)

Prepping the data

After we've taken several recordings, we want to preprocess the data. This involves creating a new project using the Classification Template (because we're just trying to find the noise, classification seems like the right type of application).

We then import the data by clicking on Data>Add Data and going to where we had all that data (I put it in a subfolder just to know where it is).

We can Redistribute Sets which allows you to assign your data for training (click that Redistribute button).

We then go to the Preprocessor and select Contextual Window (Sliding Window) and set the Length to be whatever the longest operation was. I had it at 5 seconds because of the flush time.

From here you can Create Track from Preprocessor... which then will run on your computer and preprocess each data file.

Build Preprocessor... for fun!

Now you can go back to the Data and Merge Data to get everything all together. You're now ready to try and do some Machine Learning!

Training that Machine!

This is probably the longest part of this project and frankly, the most frustrating (but you'll find out why in a bit!).

Go to Training and select Generate Model List...

Here you'll select a bunch of things that you have no real context of what any of the things are but for Auto ML, I just left it all as default (Model Size Small didn't seem to help so leave it Medium).

Within Training there's an option for Batch Size and Epochs. The default is Batch Size of 1 (meaning the number of data samples used in one forward/backward pass of training before updating the model’s weights) and Epoch is how many times you're going to iterate over the data. It really wants you to do 100 Epochs which will literally take FOREVER so I lowered it down to 50 and then later lowered it down even more.

From there you can start a New Training Job and check off Generate Labels so you can see what it's doing.

It'll start sending it to the cloud and then you can look at it later.

You can watch the progress or just close out of it and then look after a couple of hours when it seems like it's done.

Getting the Expected Results

If everything went well, you should be able to gaze upon the algorithm that THE MACHINES have created for you.

So the job will start.

The job will finish.

And then you can gaze upon your results. I took like 5 audio files where each one was several minutes long and I annotated when I flushed and when I turned on the sink. Overall there were like 50 or so datapoints for the algorithm to be trained on which I figured was enough. If you look at the job results though, it doesn't predict anything because the algorithm thinks that if it just says that there was no flush then it would be correct most of the time. It's why you see the predicted result just being 0 all over the place.

Doing Some Things

So then I decided "hey, maybe I don't have Enough data points - yeah that's it!" So I did that and I gathered more flush and sink noises - like a lot more and then ran it through the same process and then was annoyed by the lack of the results.

The model was just convinced that it was easier to say that nothing happened because statistically, if I just say that there was no noise, I would be right the vast majority of the time.

Narrowing the Scope

I think there were a few reasons why it didn't work my first few attempts and those boiled down to:

1) Too many things I wanted to train on - sink and flush. Sink noises are shorter so let's focus on that.

2) Too long of recordings - instead of minute long recordings, let's just make ten second recordings where the sink is on for more than half the time.

So I rejected all the recordings I did before and focused on just sink recordings and made them all really short. This resulted in a lot more files that I had to preprocess but overall the time was about the same. I felt like the flush noise was too long so maybe I'll address that later. It's always good to narrow your scope to the point where you're happy.

Once my table actually looked like a thing, I downloaded the result.

It was now time to actually test it out. I went back to my data and merged the conv1d data with my sessions.

I created a new file and selected Graph Unit to make an empty graph.

The Keras Model is over in the Node Explorer. The documentation makes it seem like it's on the left side but it's not!

Map it to your model.

Drag your PSOC™ 6 Artificial Intelligence Evaluation Kit (CY8CKIT-062S2-AI) Microphone so you can have it be an input to the model.

Drag a label track and a couple of data tracks so we can see it in real time.

Don't forget a Microphone Data track!

Start recording and see if it works!

I made a short video showing it working!

Going From Here...

So with all of this, the question I have for myself is "where do I go from here?" I still have to get it to do the flushing and then to do the IoT stuff afterwards. I think by narrowing the scope of "can I make a thing that detects when I use the sink?" was answered and the answer is "yes". The PSOC™ 6 Artificial Intelligence Evaluation Kit (CY8CKIT-062S2-AI) and DeepCraft does make the process relatively easy once you get the hang of what you're doing. I think that I'm going to go back to the drawing board and see if I can get the flushing to work as well. Once I do, I should be able to make that Code Gen stuff working and then maybe implement it? We'll see but this was a fun, albeit frustrating at times, experiment.