Agents are "starting to run football" and are "sucking tons of money" out of the game, according to West Ham co-chairman David Gold.
Gold was at a meeting on Thursday where Premier League clubs discussed the idea of putting a stop to agent payments.
Such a measure would mean only players would pay agents during a deal, rather than both parties.
"We are all trying to pay less - they are doing nothing for football," said Gold.
An investigation by the Times showed four out of five deals in the top flight last season involved the same agent being paid by both the player and the club.
Asked whether agents were disrupting the game, Gold added: "That goes with the territory, they are certainly doing that.
"They are starting to run football. They are certainly sucking tons of money out of football."
The Times article said top-flight clubs paid agents £211m last year, with 79% of 541 player transactions involving dual representation - when an agent acts for both the club and the player.
Meanwhile, Mino Raiola is said to have represented all three parties in Paul Pogba's £89 million move from Juventus to Manchester United in 2016, earning £41m.
"I can't believe it is even a discussion," added Gold. "Other than agents, everybody else is against them, they do nothing for football.
"The FA certainly is in line with our feelings that something has to be done. From the beginning it's not been something that we want and they don't want.
"Now it's Fifa and Uefa we need to come into line and we are hoping they come up with something."
A Football Association spokesperson told BSports: "The FA is talking to both Fifa and the Premier League and this will determine our direction of travel."
There are two ways clubs can pay agent fees, either on the player's behalf - and deducting the payment from the player's salary - or through dual representation, which club chairmen have been discussing banning, where an agent is paid by the club and the player.
For players, this means more favourable tax liabilities and for clubs it helps them secure a deal when more than one club is interested in the player. However, representing both parties gives the agent the bargaining power in the deal.
Former agent and sports law expert at Loughborough University Dr Serhat Yilmaz, who specialises on the regulation of intermediaries, says a dual representation ban is unlikely to hit the pockets of agents.
"As long as salaries are high, the fees of agents will be high," he told BSports. "Therefore, in my view, the impact of clubs not paying would not be significant.
"In terms of Premier League clubs, can the clubs become disadvantaged in a deal by not engaging with an intermediary, in particular where a number of clubs wants to sign the player? Perhaps yes.
"Nonetheless, considering that the Premier League remains the most desired league to play in by the players, it is a prestigious league for intermediaries to have their clients playing, and pays some of the highest levels of salaries in European football, I do not think the ban of dual representation would weaken the bargaining power of Premier League clubs in the player market.
"They should still be able to attract the best talent to sign for them."
Uefa and Fifa are already considering imposing a cap on agents' fees, with world football's governing body concerned about money "flowing out of the football industry".
Dr Yilmaz agrees that, to reduce the level of fees, a commission cap would be "ideal and more effective".
"The existing Fifa and the FA Intermediary Regulations do have a recommended cap on the level of commission payable to intermediaries, but it is purely recommendation and hardly complied in practice," he added.
Dr Yilmaz said overall the proposed changes, including the ban on dual representation, the re-introduction of an examination to enter into the profession an additional transparency measures, are positive and can help "bring re-order into the intermediary market following the deregulation by Fifa in 2015".