If an aluminum or brass D-pad on your Game Boy Advance doesn't feel right, below are some things you can try.  

1) Use the stock silicone if possible

2) Make sure the silicone hasn't moved out of position during assembly

3)  There are two locating holes in the silicone, and only 1 bottom locating post in the aluminum housing.  In order to compensate, you can:

       a) Add a 3d printed top locating post.  See image below.  It's the white piece designated by the red arrow.  Contact me to get one to tryout.  You need to make sure it is sitting flat.  A tip is to sand or scrape the bottom edges to ensure it fits well

       b) Cut a slit in the bottom locating hole in the silicone.  It's difficult to see if the image, however this is done to bias the D-pad up.  See green arrow pointing to the area that is cut.

4) If diagonals are an issue, you can try to round the bottom post on your aluminum/brass D-pad.  Take a known good plastic D-pad and carefully round it until it matches the shape of the plastic piece.