Drop libm from the linker dependencies for libffi