Reinforcement Learning Hexapod

[blegas78]

I had originally sent a PM to Matt about servo motor recommendations, and he gave me great advice. He then kindly suggested that I post my project, and I will gladly do so.

I am taking ECE 596c this semester, titled Cognitive Robotics. For the semester we are supposed to build a robot that deals with ideas of cognition. Most people involve human-robot social interaction or robots with emotional engines. I have always loved hexapods and began but never completed a hexapod project due to time and money. Now however, I have a perfect excuse to spend much time on my hexapod. The idea of cognition that I wanted to incorporate into my hexapod is the ability to learn. Learning is a very important part of cognition, as humans learn new skills an uncountable number of times in their life. It is the ability to learn that allows us to become great problem solvers, tool makers, and language builders.

The Project: I do not want to program any walking gait into the system using inverse kinematics or any other method. Instead, based on sensory input, the robot will figure out on it's own what the best method is to control the motors to move straight forward. I am still trying to figure out what the most efficient method will be to accomplish this, as there are many different forms of machine learning. I have many ideas, I just need to figure out which method I will use and stick with it.

Mechanics: I started some of it 5 years ago, but I am rebuilding most of it now. I never had a lot of money, so I tried to get by in the cheapest way possible. The attached pictures show a second rebuild of the project that I did after my first year of college. I had the money to buy some of the servo erector set pieces from Lynxmotion which really cleaned up the appearance a lot. I didn't go all out though, and I purchased some pieces of brass tubing, angle aluminum, and thread rod to build the femur and tibia. This was much cheaper than ordering anything off the internet.

Servos: Pictured, the coxa servos are futaba s3003's which is ok, femur servos are futaba s3305's which are current hungry but provide plenty of torque, and the tibia servos as again futaba s3003's which are quite weak for the job. Having sent Matt a PM, he recommended dynamixel AX-12s. I was little scared of using these at first, since I only wanted to buy six PWM based motors as I already had Pololu serial servo controllers. I decided to get one and see how I liked it. I am very impressed with them, especially from a research robotics point of view. The plan is now to move the s3305 servos to the tibia, and have new AX-12 servos at the femur. I went ahead and purchased another five, and am in the process of interfacing both the AX-12 motors and Pololu controllers to a single PIC, so that I can control them both through the same serial line. It will certainly be an ugly mess of code...

Electronics/Software: As mentioned above, there will be a PIC that interfaces to the AX-12s and the Pololu controllers. For all of the control, I will use MATLAB, as it is very powerful, and I can use all of the resources of any powerful computer. I have a BlueSmirf that will interface the PIC to a computer. The nice thing about using a PIC as opposed to say a purchased dynamixel-USB/Serial converter is that I can very easily interface any sensor I want to the PIC and get that information sent to MATLAB. I was inspired by Matt's V4 and V5 I think with the force compliance, and so I have been working on adding buttons to the legs of my hexapod. I will post images when I finish. I will also include an accelerometer (LIS302DL) if I can get the thing to communicate to the PIC. I also need some sort of information where I get feedback of how well the hexapod moves forward based on leg motion experimentation. I will have to incorporate a camera somehow I think...

Here's a couple quick shots of what I have so far. I will post updates as they come. Any comments are welcome

[Noodle]

Very exciting! Looking forward to see a learning hexapod!

[Matt Denton]

Great start to your hexapod project.. it looks like a big beast!

I look forward to seeing it learn to walk

[TXBDan]

Very cool, good luck.

btw, where are you going to school?

[wuselfuzz]

Wow, slick design. Simple, yet elegant.

I also thought about letting some machine learning algorithms loose on my pod.

However, to do any of that, I need sensors first which is the current challenge.

One idea I had: add current sensors to the power wires to each servo, this would give an
approximation of abs(torque). Then, the bot could try to even out individual servo load while
also minimizing the total servo load and (if a sensor to measure that is available) to keep a constant
height above the ground.

I'm also more interested in carrying necessary CPU power on board, so while modeling an algorithm in MATLAB for test purposes is fine, I'd rather see an efficient implementation in C as a final result.

[florek]

Pictures doesn't work. Nice project. Sounds like something similar to Starfish self modelling robot form MIT(?)

[blegas78]

Thanks for the comments!

Great start to your hexapod project.. it looks like a big beast!

I look forward to seeing it learn to walk

I too think that this is a big beast, but I actually tend to like the smaller hexapods! I must say that the uBug looks pretty appealing...

btw, where are you going to school?

I am currently attending the University of Arizona. I am very glad that there are a few robotics classes, but there are only a few.

Wow, slick design. Simple, yet elegant.

I don't have any CNC machines or laser cutters. All I have is a hack saw, file, soldering iron, and drill press. Hopefully I can inspire other people to build little machines without having access to nicer equipment.

One idea I had: add current sensors to the power wires to each servo, this would give an
approximation of abs(torque).

Though I could be mistaken, I believe this is how the AX-12 motors sense "torque."

I'm also more interested in carrying necessary CPU power on board, so while modeling an algorithm in MATLAB for test purposes is fine, I'd rather see an efficient implementation in C as a final result.

I would definitely like to do this as well, so the robot would be truly mobile. Unfortunately, The amount of memory needed for the project could be very excessive, and the resources of a desktop computer are fantastic. If I get something efficient, then maybe I will incorporate this into a custom mobile microprocessing platform.

[Mark H]

Get yourself a VIA EPIA PX10000G, 10cm x 7cm - takes a single SO-DIMM upto 1GB.

I built a neural net bot a few years back for World of Warcraft, it used to run at about 2.5gb of RAM with C#, and optimisations for speed rather than memory usage. Making something that just runs a bot on 1GB of ram should be pretty easy, especially with C.

[blegas78]

over the last two days I have been adding force biased buttons to the tibia, as I originally stated is inspired by Matt's force compliance hexapods. I also received all of the AX-12 motors yesterday, so I finished installing them. I ran out of heat shrink, so I haven't built the entire thing yet, but here is a quick shot of the completed legs:



Also here is a quick shot of the tibia. A light spring is used to hold the piece of brass tubing. A heavier, shorter spring provides a very slight amount of force biasing, meaning that a certain amount of force is needed in order to push the button. I had to cut the heavier springs myself from a longer length spring. The three washers between the spring and tubing allow for a little bit of calibration. Small, bent copper tubing is soldered to the end of the brass tubing. One piece clearly touches the button when a certain amount of force is applied to the tibia. Two other identical pieces of tubing help prevent the brass tubing from twisting. One piece cannot be seen as it is on the other side of the first piece. There is a little slop between the thread-rod and the tubing, but not enough to vastly affect anything, even if inverse kinematics were applied to the robot.

Get yourself a VIA EPIA PX10000G, 10cm x 7cm - takes a single SO-DIMM upto 1GB.

I built a neural net bot a few years back for World of Warcraft, it used to run at about 2.5gb of RAM with C#, and optimisations for speed rather than memory usage. Making something that just runs a bot on 1GB of ram should be pretty easy, especially with C.

Thanks for the info, I will definitely look into this!

[Matt Denton]

Work is progressing well with your hex! Interesting tibia contact design, how well does it perform at non perpendicular angles from the ground?

Let me know how the AX-12's perform, I have been tinkering with an RX-10 and an RX-28 lately, very nice servos!