Reinforcement Learning Hexapod

[wuselfuzz]

I would definitely like to do this as well, so the robot would be truly mobile. Unfortunately, The amount of memory needed for the project could be very excessive, and the resources of a desktop computer are fantastic. If I get something efficient, then maybe I will incorporate this into a custom mobile microprocessing platform.

I pondered about getting an eddy ARM CPU module: http://www.embeddedmodule.com/

An external CPU would save a lot of FPGA resources.

- 32-bit ARM9 CPU / 8MB Flash / 32MB SDRAM
- Pin Header Interface(144 pin)
- 10/100 Ethernet PHY (Auto MDIX) & 2 UARTs
- Max 56 Programmable GPIO
- 4 channel UART
- TWI(I2C),SPI,MCI,ADC
- 2 USB Host, 1 USB Device
- Supported by Dev Kit including SDK, API
- Operated by Real Time Linux, Lemonix
- Supported by Eclipse based IDE, LemonIDE
- Provides easy-to-use Windows utilities
COM Port Redirector, PortView, TestView
- Watchdog timer supported
- Compact sized to fit into any hardware
- Operation Temp : -40 ~ 85 C

The module is 25x48.5mm large and only weights a few grams.

The CPU is running at 210 MHz and 32 MB ram is a good start. The slightly older eddy v2.0 with 4MB flash/32MB RAM/180 MHz CPU is available for 40 euros, not including tax.

[Matt Denton]

That's a really nice little board, and looks pretty powerful! do you know what kind of MIPS rating those processor would have?

[wuselfuzz]

do you know what kind of MIPS rating those processor would have?

MIPS ratings are only as good as you fake them. For microcontrollers, it's often taken as literal as possible, describing the number of assembler instructions per second they can process. But then, given C code can perform totally different depending on the underlying architecture.

An example:

Code: Select all

int add(int a, int b) {
   return a+b;
}


This compiles (with -Os to optimize for size) to this code on MIPS:

Code: Select all

Disassembly of section .text:

00000000 <add>:
   0:   03e00008    jr   ra
   4:   00a41021    addu   v0,a1,a0


It's no bug that the add command follows the jump to the return address saved in register $ra, that's actually a feature of MIPS out of order execution. If neccessary, the compiler would fill this delayed execution slot with a nop.

This compiles to this code for AVR microcontollers:

Code: Select all

Disassembly of section .text:

00000000 <add>:
   0:   86 0f          add   r24, r22
   2:   97 1f          adc   r25, r23
   4:   08 95          ret


As you can see, an additional instruction is neccessary, because the AVR can only add one register to another register and not add one register to another and store the result to a third register. Additionally, the registers on an atmel are 8 bits while the registers on a MIPS are 32 bits wide.

Then, if you're doing complex calculations (e.g inverse kinematics), a trivial implementation using floats would be much, much faster if there's an FPU available.

Random link to a writeup that hits the nail on the head: http://www.futuretech.blinkenlights.nl/perf.html

/edit: I'd think the ARM CPU at ~200 MHz would be about four times as fast as my soft-CPU and much faster than any microcontroller, on-the-fly IK calculations should work like a charm on that. You could also think of it being three times as fast as the Nintendo DS's main cpu or an old Playstation 1. But still, comparing processors is a bit hard, to give one of those dishonorable car analogies, comparing a microcontroller to an embedded CPU and a server CPU is like comparing a go cart to a compact car and a lorrytruck (not being either british or american sometimes confuses my english skills). Raw horsepower says nothing about max speed and suitability for your application, i.e. the truck might have a much larger value for its horsepower but still might be slower. On the other hand, try to transport a truckload of washing machines with your compact car...

/edit2: I just fired up DSlinux on my DS and that reports 33 BogoMIPS (the linux developers called that BogoMIPS for a reason) for the 66 MHz ARM9

[blegas78]

Work is progressing well with your hex! Interesting tibia contact design, how well does it perform at non perpendicular angles from the ground?

Let me know how the AX-12's perform, I have been tinkering with an RX-10 and an RX-28 lately, very nice servos!

The contact buttons certainly don't perform nearly as well at non-perpendicular angles, which is not necessarily a bad thing for my purposes. As you know, If the robot walked with many of the tibias at extreme angles, this would cause a lot of strain on the tibia servos. Also, if I have this setup, then I constrain the system a little more, since the only sensed inputs are when the legs are in a more "natural" position. It's definitely a good concern, and hopefully everything will work ok for me.

So far I really like the servos! I like the ability to set the speed and maximum torque, as well as the ability to get the feedback information. They are not as easy to get up and running, but they are by no means too difficult. Even if they do not perform quite as well torque-wise to other servos, they are awesome when it comes to true robotics.

I did build an IK function in matlab and also a quick walking algorithm so when I get it all connected and running, I will try to post a video.

[blegas78]

One thing I forgot to mention in the above post about the AX-12 motors is the range. I really like the 300 degree range, and also that they can be easily electronically converted into continuous rotation, simply by setting the position limit registers to zero or something. Here is a shot of the mechanical part of the hexapod complete. There's a real fun mess of wires coming out through the center:



Here's another shot of the ugly prototype electronics I very poorly mounted. The next step is to rebuild part of the AX-12 - Matlab PIC interface to work with the Pololu serial controller, then power up it up and get it walking.

[blegas78]

Well earlier this week I was calibrating the Futaba servos. I had an equation error in my Matlab code that resulted in sending a position value to a servo, resulting in setting one of the S3305 servos to a position beyond its range. It still tried to get there, but the position was beyond the physical limit of the servo. Sitting there for about a minute, I figured out the miscalculation, then tried to set the servo to a position in its range. Now it didn't budge! I disconnected the power and tried to move the servo horn by hand. I tried as hard as I could to move it, but it was stuck. I took the motor apart and found that the motor was locked up. I guess I need a new servo!

So what do I do, get another servo like this one or upgrade? My philosophy is to upgrade when parts break, so I ordered an RX-28 to replace it. Now I will have the RX-28 as the femur servo, and the AX-12 that was the femur servo will be moved to the tibia. A little uneven? Definitely, but at least it will perform better. Eventually I would like to get RX-28 servos for every femur on the robot, so this is a small step to get there.

I have also been working on a way to measure forward motion using optical flow. I purchased a Logitech Quickcam to mount on the robot. I have some optical flow code in the works. Pictures to come soon hopefully....

[blegas78]

Now my hex has a face!

I picked up a Logitech Quickcam Communicate Deluxe and mounted it on the front of the hex. I have been working on getting some optical flow code working to measure total forward movement. So far it has been working fairly well, but does need a little bit of error correction.



Also, I got the RX-28 up and running. Now I have three types of servos with different communication needs, and also a hexapod leg that's a little beefier than the rest...



I also have my button feedback working fairly well. I shall post a video hopefully shortly...

[blegas78]

Well before I post a video, here's a quick demonstration of software:

In order to get the hexapod to learn to walk, I need to actually get it to walk under my own hard coded algorithms first, to make sure the hardware works. Here, I have Matlab running a inverse kinematic engine to map out a simple trajectory for each leg. I have it make a quick plot every time the hexapod gets updated. Here is a quick shot of the simulation:


The servo angles are calculated and sent to the hexapod, and the hexapod gives button feedback from each leg. If the plot shows blue, it means that the leg is within its possible range, but the button is not in contact with anything. If a part of the leg is red, it means the corresponding servo is out of range for the desired X,Y,Z position. Here you can see that the one leg has the femur as red, meaning that the femur servo (the RX-28 in this case) is beyond the physical range. If the tibia is green, then the button is making contact with the ground. This plot shows the hex walking in the positive y direction.

In terms of other feedback, I have optical flow running, which doesn't work with Matlab as of now but will soon:


So this image is the program getting video back from the Quickcam with the hexapod walking forward. The program essentially takes a snapshot, finds good features in the image, then takes another snapshot and does its best to find the exact same features in the second image. The program then draws the corresponding vectors on the second image. I have the program also draw three other vectors:

1.) In the black part of the image, there is a tiny line to the right of the image, pointing up and to the left. This is a vector that is the negative sum of all the vectors, then is normalized by dividing the sum by the total number of vectors (in other words the average vector in the opposite direction). This vector gives translation information from the image. In the bottom right of the image, you see an object that moved relatively fast in the field of view, creating larger vectors, which results in this tiny vector pointing in the opposite direction. This means the hexapod moved the camera up and to the left a little bit.

2.) Below and between the image and the above translation vector, there are two vectors. The vector pointing up and down averages the dot product of all of the vectors with the location of the vector from the center of the image. The dot product results in a scalar value, not a vector, so I simply build a vector that points upward that has the magnitude of the dot product. If the robot moves straight forward, then the vectors of the image will all point outward from the center of the image. This results in a higher value for the dot product, resulting in a higher positive numerical "forward movement" value. If the hexapod were to move straight backward, then the vector field would point more towards the center of the image, resulting in a lower negative numerical "backward movement" value.

3.) The vector that is located at the same initial position as the above vector, but pointing horizontally is the average cross product value. Obviously I cannot point a vector out of or into the screen which would be the true cross product result, so I simply have it point the magnitude horizontally. Again, the cross product is calculated for every optic flow vector crossed with its vectorial location with respect to the center of the image. The more perpendicular the two vectors are, the higher the value of the cross product. If the hexapod is to roll slightly left or right, then the average cross product would result in positive or negative values. This one is difficult to see from this image, but since there is a clear value of the average cross product, the hexapod did roll a little bit when moving forward.

There are some dislikes about the optical flow code. Though I like that the code finds the best features in the image for optical flow accuracy, I would prefer that the code find features of the image more evenly distributed in the image. Since the program gets more vectors in a localized area, this greatly throws off the balance of the image, and can therefore heavily distort the data. I am not an expert at this stuff, so I will ask the main coder of the optic flow code to see if I can make a quick solution...

[Matt Denton]

All sounds pretty complex, and looks like you have a pretty solid plan.. but the images aren't working for me. I did check the embedded link, but can't get it to work, do you have another place you can host them?

Keep up the hard work!

[blegas78]

Hmm, for some reason the server dropped immediately after I had posted the pictures, but the server usually never has issues.

Anyway they should be up now...